standard_metrics#

Standard machine learning objective functions.

Module Contents#

Classes Summary#

AccuracyBinary

Accuracy score for binary classification.

AccuracyMulticlass

Accuracy score for multiclass classification.

AUC

AUC score for binary classification.

AUCMacro

AUC score for multiclass classification using macro averaging.

AUCMicro

AUC score for multiclass classification using micro averaging.

AUCWeighted

AUC Score for multiclass classification using weighted averaging.

BalancedAccuracyBinary

Balanced accuracy score for binary classification.

BalancedAccuracyMulticlass

Balanced accuracy score for multiclass classification.

ExpVariance

Explained variance score for regression.

F1

F1 score for binary classification.

F1Macro

F1 score for multiclass classification using macro averaging.

F1Micro

F1 score for multiclass classification using micro averaging.

F1Weighted

F1 score for multiclass classification using weighted averaging.

Gini

Gini coefficient for binary classification.

LogLossBinary

Log Loss for binary classification.

LogLossMulticlass

Log Loss for multiclass classification.

MAE

Mean absolute error for regression.

MAPE

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

MaxError

Maximum residual error for regression.

MCCBinary

Matthews correlation coefficient for binary classification.

MCCMulticlass

Matthews correlation coefficient for multiclass classification.

MeanSquaredLogError

Mean squared log error for regression.

MedianAE

Median absolute error for regression.

MSE

Mean squared error for regression.

Precision

Precision score for binary classification.

PrecisionMacro

Precision score for multiclass classification using macro-averaging.

PrecisionMicro

Precision score for multiclass classification using micro averaging.

PrecisionWeighted

Precision score for multiclass classification using weighted averaging.

R2

Coefficient of determination for regression.

Recall

Recall score for binary classification.

RecallMacro

Recall score for multiclass classification using macro averaging.

RecallMicro

Recall score for multiclass classification using micro averaging.

RecallWeighted

Recall score for multiclass classification using weighted averaging.

RootMeanSquaredError

Root mean squared error for regression.

RootMeanSquaredLogError

Root mean squared log error for regression.

Contents#

class evalml.objectives.standard_metrics.AccuracyBinary[source]#

Accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Accuracy Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for accuracy score for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for accuracy score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.AccuracyMulticlass[source]#

Accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Accuracy Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for accuracy score for multiclass classification.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for accuracy score for multiclass classification.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUC[source]#

AUC score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for AUC score for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for AUC score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.AUCMacro[source]#

AUC score for multiclass classification using macro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for AUC score for multiclass classification using macro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for AUC score for multiclass classification using macro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUCMicro[source]#

AUC score for multiclass classification using micro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for AUC score for multiclass classification using micro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for AUC score for multiclass classification using micro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUCWeighted[source]#

AUC Score for multiclass classification using weighted averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.1, 0.2, 0.7],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.2, 0.3]]
>>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

AUC Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for AUC Score for multiclass classification using weighted averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for AUC Score for multiclass classification using weighted averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.BalancedAccuracyBinary[source]#

Balanced accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Balanced Accuracy Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for accuracy score for balanced accuracy for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for accuracy score for balanced accuracy for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.BalancedAccuracyMulticlass[source]#

Balanced accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Balanced Accuracy Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for accuracy score for balanced accuracy for multiclass classification.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for accuracy score for balanced accuracy for multiclass classification.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.ExpVariance[source]#

Explained variance score for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736)

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

ExpVariance

perfect_score

1.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for explained variance score for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for explained variance score for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1[source]#

F1 score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for F1 score for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for F1 score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.F1Macro[source]#

F1 score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for F1 score for multiclass classification using macro averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for F1 score for multiclass classification using macro averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1Micro[source]#

F1 score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for F1 score for multiclass classification.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for F1 score for multiclass classification.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1Weighted[source]#

F1 score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

F1 Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for F1 score for multiclass classification using weighted averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for F1 score for multiclass classification using weighted averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Gini[source]#

Gini coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

Gini

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for Gini coefficient for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for Gini coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.LogLossBinary[source]#

Log Loss for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 19.6601745)

Attributes

expected_range

[0, 1]

greater_is_better

False

is_bounded_like_percentage

False

name

Log Loss Binary

perfect_score

0.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for log loss for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for log loss for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.LogLossMulticlass[source]#

Log Loss for multiclass classification.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301)

Attributes

expected_range

[0, 1]

greater_is_better

False

is_bounded_like_percentage

False

name

Log Loss Multiclass

perfect_score

0.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

True

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for log loss for multiclass classification.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for log loss for multiclass classification.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MAE[source]#

Mean absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

True

name

MAE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for mean absolute error for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for mean absolute error for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MAPE[source]#

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Mean Absolute Percentage Error

perfect_score

0.0

problem_types

[ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for mean absolute percentage error for time series regression.

positive_only

If True, this objective is only valid for positive data.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for mean absolute percentage error for time series regression.

positive_only(self)#

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MaxError[source]#

Maximum residual error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MaxError

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for maximum residual error for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for maximum residual error for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MCCBinary[source]#

Matthews correlation coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457)

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

MCC Binary

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for Matthews correlation coefficient for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for Matthews correlation coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.MCCMulticlass[source]#

Matthews correlation coefficient for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325)

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

MCC Multiclass

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for Matthews correlation coefficient for multiclass classification.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for Matthews correlation coefficient for multiclass classification.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MeanSquaredLogError[source]#

Mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Mean Squared Log Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for mean squared log error for regression.

positive_only

If True, this objective is only valid for positive data.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for mean squared log error for regression.

positive_only(self)#

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MedianAE[source]#

Median absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MedianAE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for median absolute error for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for median absolute error for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MSE[source]#

Mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

MSE

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for mean squared error for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for mean squared error for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Precision[source]#

Precision score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for precision score for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for precision score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.PrecisionMacro[source]#

Precision score for multiclass classification using macro-averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for precision score for multiclass classification using macro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for precision score for multiclass classification using macro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.PrecisionMicro[source]#

Precision score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for precision score for binary classification using micro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for precision score for binary classification using micro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.PrecisionWeighted[source]#

Precision score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Precision Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for precision score for multiclass classification using weighted averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for precision score for multiclass classification using weighted averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.R2[source]#

Coefficient of determination for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036)

Attributes

expected_range

None

greater_is_better

True

is_bounded_like_percentage

False

name

R2

perfect_score

1

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for coefficient of determination for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for coefficient of determination for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Recall[source]#

Recall score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for recall score for binary classification.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for recall score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.

class evalml.objectives.standard_metrics.RecallMacro[source]#

Recall score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Macro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for recall score for multiclass classification using macro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for recall score for multiclass classification using macro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RecallMicro[source]#

Recall score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Micro

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for recall score for multiclass classification using micro-averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for recall score for multiclass classification using micro-averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RecallWeighted[source]#

Recall score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Recall Weighted

perfect_score

1.0

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for recall score for multiclass classification using weighted averaging.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for recall score for multiclass classification using weighted averaging.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RootMeanSquaredError[source]#

Root mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Root Mean Squared Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for root mean squared error for regression.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for root mean squared error for regression.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RootMeanSquaredLogError[source]#

Root mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204)

Attributes

expected_range

None

greater_is_better

False

is_bounded_like_percentage

False

name

Root Mean Squared Log Error

perfect_score

0.0

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Objective function for root mean squared log error for regression.

positive_only

If True, this objective is only valid for positive data.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#

Objective function for root mean squared log error for regression.

positive_only(self)#

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters
  • y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

  • y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.