Objectives

Package Contents

Classes Summary

AccuracyBinary

Accuracy score for binary classification.

AccuracyMulticlass

Accuracy score for multiclass classification.

AUC

AUC score for binary classification.

AUCMacro

AUC score for multiclass classification using macro averaging.

AUCMicro

AUC score for multiclass classification using micro averaging.

AUCWeighted

AUC Score for multiclass classification using weighted averaging.

BalancedAccuracyBinary

Balanced accuracy score for binary classification.

BalancedAccuracyMulticlass

Balanced accuracy score for multiclass classification.

BinaryClassificationObjective

Base class for all binary classification objectives.

CostBenefitMatrix

Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric

ExpVariance

Explained variance score for regression.

F1

F1 score for binary classification.

F1Macro

F1 score for multiclass classification using macro averaging.

F1Micro

F1 score for multiclass classification using micro averaging.

F1Weighted

F1 score for multiclass classification using weighted averaging.

FraudCost

Score the percentage of money lost of the total transaction amount process due to fraud.

Gini

Gini coefficient for binary classification.

LeadScoring

Lead scoring.

LogLossBinary

Log Loss for binary classification.

LogLossMulticlass

Log Loss for multiclass classification.

MAE

Mean absolute error for regression.

MAPE

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

MaxError

Maximum residual error for regression.

MCCBinary

Matthews correlation coefficient for binary classification.

MCCMulticlass

Matthews correlation coefficient for multiclass classification.

MeanSquaredLogError

Mean squared log error for regression.

MedianAE

Median absolute error for regression.

MSE

Mean squared error for regression.

MulticlassClassificationObjective

Base class for all multiclass classification objectives.

ObjectiveBase

Base class for all objectives.

Precision

Precision score for binary classification.

PrecisionMacro

Precision score for multiclass classification using macro averaging.

PrecisionMicro

Precision score for multiclass classification using micro averaging.

PrecisionWeighted

Precision score for multiclass classification using weighted averaging.

R2

Coefficient of determination for regression.

Recall

Recall score for binary classification.

RecallMacro

Recall score for multiclass classification using macro averaging.

RecallMicro

Recall score for multiclass classification using micro averaging.

RecallWeighted

Recall score for multiclass classification using weighted averaging.

RegressionObjective

Base class for all regression objectives.

RootMeanSquaredError

Root mean squared error for regression.

RootMeanSquaredLogError

Root mean squared log error for regression.

SensitivityLowAlert

Base class for all binary classification objectives.

Functions

get_all_objective_names

Get a list of the names of all objectives.

get_core_objective_names

Get a list of all valid core objectives.

get_core_objectives

Returns all core objective instances associated with the given problem type.

get_non_core_objectives

Get non-core objective classes.

get_objective

Returns the Objective class corresponding to a given objective name.

Contents

class evalml.objectives.AccuracyBinary[source]

Accuracy score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Accuracy Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AccuracyMulticlass[source]

Accuracy score for multiclass classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Accuracy Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUC[source]

AUC score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCMacro[source]

AUC score for multiclass classification using macro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCMicro[source]

AUC score for multiclass classification using micro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCWeighted[source]

AUC Score for multiclass classification using weighted averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BalancedAccuracyBinary[source]

Balanced accuracy score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Balanced Accuracy Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BalancedAccuracyMulticlass[source]

Balanced accuracy score for multiclass classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Balanced Accuracy Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BinaryClassificationObjective[source]

Base class for all binary classification objectives.

Attributes

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

expected_range

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

greater_is_better

Returns a boolean determining if a greater score indicates better model performance.

is_bounded_like_percentage

Returns whether this objective is bounded between 0 and 1, inclusive.

is_defined_for_problem_type

name

Returns a name describing the objective.

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

perfect_score

Returns the score obtained by evaluating this objective on a perfect model.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

score_needs_proba

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)[source]

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)
property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)[source]

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)[source]

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.CostBenefitMatrix(true_positive, true_negative, false_positive, false_negative)[source]

Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.

Parameters
  • true_positive (float) – Cost associated with true positive predictions

  • true_negative (float) – Cost associated with true negative predictions

  • false_positive (float) – Cost associated with false positive predictions

  • false_negative (float) – Cost associated with false negative predictions

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

Cost Benefit Matrix

perfect_score

None

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Calculates cost-benefit of the using the predicted and true values.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Calculates cost-benefit of the using the predicted and true values.

Parameters
  • y_predicted (pd.Series) – Predicted labels

  • y_true (pd.Series) – True labels

  • X (pd.DataFrame) – Ignored.

  • sample_weight (pd.DataFrame) – Ignored.

Returns

Cost-benefit matrix score

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.ExpVariance[source]

Explained variance score for regression.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

ExpVariance

perfect_score

1.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1[source]

F1 score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Macro[source]

F1 score for multiclass classification using macro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Micro[source]

F1 score for multiclass classification using micro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Weighted[source]

F1 score for multiclass classification using weighted averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.FraudCost(retry_percentage=0.5, interchange_fee=0.02, fraud_payout_percentage=1.0, amount_col='amount')[source]

Score the percentage of money lost of the total transaction amount process due to fraud.

Parameters
  • retry_percentage (float) – What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to 0.5.

  • interchange_fee (float) – How much of each successful transaction you pay. Between 0 and 1. Defaults to 0.02.

  • fraud_payout_percentage (float) – Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0.

  • amount_col (str) – Name of column in data that contains the amount. Defaults to “amount”.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

True

name

Fraud Cost

perfect_score

0.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X, sample_weight=None)[source]

Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.

Parameters
  • y_predicted (pd.Series) – Predicted fraud labels

  • y_true (pd.Series) – True fraud labels

  • X (pd.DataFrame) – Data with transaction amounts

  • sample_weight (pd.DataFrame) – Ignored.

Returns

Amount lost to fraud per transaction

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

evalml.objectives.get_all_objective_names()[source]

Get a list of the names of all objectives.

Returns

Objective names

Return type

list (str)

evalml.objectives.get_core_objective_names()[source]

Get a list of all valid core objectives.

Returns

Objective names.

Return type

list[str]

evalml.objectives.get_core_objectives(problem_type)[source]

Returns all core objective instances associated with the given problem type.

Core objectives are designed to work out-of-the-box for any dataset.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

evalml.objectives.get_non_core_objectives()[source]

Get non-core objective classes.

Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives before using them in AutoMLSearch.

Returns

List of ObjectiveBase classes

evalml.objectives.get_objective(objective, return_instance=False, **kwargs)[source]

Returns the Objective class corresponding to a given objective name.

Parameters
  • objective (str or ObjectiveBase) – Name or instance of the objective class.

  • return_instance (bool) – Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments.

  • kwargs (Any) – Any keyword arguments to pass into the objective. Only used when return_instance=True.

Returns

ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned.

class evalml.objectives.Gini[source]

Gini coefficient for binary classification.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

Gini

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.LeadScoring(true_positives=1, false_positives=- 1)[source]

Lead scoring.

Parameters
  • true_positives (int) – Reward for a true positive. Defaults to 1.

  • false_positives (int) – Cost for a false positive. Should be negative. Defaults to -1.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

Lead Scoring

perfect_score

None

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Calculate the profit per lead.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Calculate the profit per lead.

Parameters
  • y_predicted (pd.Series) – Predicted labels

  • y_true (pd.Series) – True labels

  • X (pd.DataFrame) – Ignored.

  • sample_weight (pd.DataFrame) – Ignored.

Returns

Profit per lead

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.LogLossBinary[source]

Log Loss for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

False

is_bounded_like_percentage

False

name

Log Loss Binary

perfect_score

0.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.LogLossMulticlass[source]

Log Loss for multiclass classification.

Attributes

expected_range

[0, 1]

greater_is_better

False

is_bounded_like_percentage

False

name

Log Loss Multiclass

perfect_score

0.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MAE[source]

Mean absolute error for regression.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

True

name

MAE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MAPE[source]

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Mean Absolute Percentage Error

perfect_score

0.0

problem_types

[ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(self)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MaxError[source]

Maximum residual error for regression.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MaxError

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MCCBinary[source]

Matthews correlation coefficient for binary classification.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

MCC Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MCCMulticlass[source]

Matthews correlation coefficient for multiclass classification.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

MCC Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MeanSquaredLogError[source]

Mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Mean Squared Log Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(self)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MedianAE[source]

Median absolute error for regression.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MedianAE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MSE[source]

Mean squared error for regression.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MSE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MulticlassClassificationObjective[source]

Base class for all multiclass classification objectives.

Attributes

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

Methods

calculate_percent_difference

Calculate the percent difference between scores.

expected_range

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

greater_is_better

Returns a boolean determining if a greater score indicates better model performance.

is_bounded_like_percentage

Returns whether this objective is bounded between 0 and 1, inclusive.

is_defined_for_problem_type

name

Returns a name describing the objective.

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

perfect_score

Returns the score obtained by evaluating this objective on a perfect model.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

score_needs_proba

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)
property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.ObjectiveBase[source]

Base class for all objectives.

Attributes

problem_types

None

Methods

calculate_percent_difference

Calculate the percent difference between scores.

expected_range

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

greater_is_better

Returns a boolean determining if a greater score indicates better model performance.

is_bounded_like_percentage

Returns whether this objective is bounded between 0 and 1, inclusive.

is_defined_for_problem_type

name

Returns a name describing the objective.

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

perfect_score

Returns the score obtained by evaluating this objective on a perfect model.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

score_needs_proba

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)[source]

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)[source]
property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)[source]

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.Precision[source]

Precision score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionMacro[source]

Precision score for multiclass classification using macro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionMicro[source]

Precision score for multiclass classification using micro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionWeighted[source]

Precision score for multiclass classification using weighted averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.R2[source]

Coefficient of determination for regression.

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

R2

perfect_score

1

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.Recall[source]

Recall score for binary classification.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallMacro[source]

Recall score for multiclass classification using macro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallMicro[source]

Recall score for multiclass classification using micro averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallWeighted[source]

Recall score for multiclass classification using weighted averaging.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RegressionObjective[source]

Base class for all regression objectives.

Attributes

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

Methods

calculate_percent_difference

Calculate the percent difference between scores.

expected_range

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

greater_is_better

Returns a boolean determining if a greater score indicates better model performance.

is_bounded_like_percentage

Returns whether this objective is bounded between 0 and 1, inclusive.

is_defined_for_problem_type

name

Returns a name describing the objective.

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

perfect_score

Returns the score obtained by evaluating this objective on a perfect model.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

score_needs_proba

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)
property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RootMeanSquaredError[source]

Root mean squared error for regression.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Root Mean Squared Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RootMeanSquaredLogError[source]

Root mean squared log error for regression.

Only valid for nonnegative inputs.Otherwise, will throw a ValueError.

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Root Mean Squared Log Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Arguments:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

positive_only(self)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.SensitivityLowAlert(alert_rate=0.01)[source]

Base class for all binary classification objectives.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Sensitivity at Low Alert Rates

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Determine if an observation is high risk given an alert rate

is_defined_for_problem_type

objective_function

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Default False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, **kwargs)[source]

Determine if an observation is high risk given an alert rate

Parameters

ypred_proba (pd.Series) – Predicted probabilities

classmethod is_defined_for_problem_type(cls, problem_type)
objective_function(self, y_true, y_predicted, **kwargs)[source]

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class

Parameters
  • y_true (pd.Series) – True labels

  • y_predicted (pd.Series) – Predicted labels based on alert_rate

Returns

sensitivity using the observations with the top scores as the predicted positive class

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)

If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None