standard_metrics#

Standard machine learning objective functions.

Module Contents#

Classes Summary#

`AccuracyBinary`	Accuracy score for binary classification.
`AccuracyMulticlass`	Accuracy score for multiclass classification.
`AUC`	AUC score for binary classification.
`AUCMacro`	AUC score for multiclass classification using macro averaging.
`AUCMicro`	AUC score for multiclass classification using micro averaging.
`AUCWeighted`	AUC Score for multiclass classification using weighted averaging.
`BalancedAccuracyBinary`	Balanced accuracy score for binary classification.
`BalancedAccuracyMulticlass`	Balanced accuracy score for multiclass classification.
`ExpVariance`	Explained variance score for regression.
`F1`	F1 score for binary classification.
`F1Macro`	F1 score for multiclass classification using macro averaging.
`F1Micro`	F1 score for multiclass classification using micro averaging.
`F1Weighted`	F1 score for multiclass classification using weighted averaging.
`Gini`	Gini coefficient for binary classification.
`LogLossBinary`	Log Loss for binary classification.
`LogLossMulticlass`	Log Loss for multiclass classification.
`MAE`	Mean absolute error for regression.
`MAPE`	Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.
`MASE`	Mean absolute scaled error for time series regression.
`MaxError`	Maximum residual error for regression.
`MCCBinary`	Matthews correlation coefficient for binary classification.
`MCCMulticlass`	Matthews correlation coefficient for multiclass classification.
`MeanSquaredLogError`	Mean squared log error for regression.
`MedianAE`	Median absolute error for regression.
`MSE`	Mean squared error for regression.
`Precision`	Precision score for binary classification.
`PrecisionMacro`	Precision score for multiclass classification using macro-averaging.
`PrecisionMicro`	Precision score for multiclass classification using micro averaging.
`PrecisionWeighted`	Precision score for multiclass classification using weighted averaging.
`R2`	Coefficient of determination for regression.
`Recall`	Recall score for binary classification.
`RecallMacro`	Recall score for multiclass classification using macro averaging.
`RecallMicro`	Recall score for multiclass classification using micro averaging.
`RecallWeighted`	Recall score for multiclass classification using weighted averaging.
`RootMeanSquaredError`	Root mean squared error for regression.
`RootMeanSquaredLogError`	Root mean squared log error for regression.
`SMAPE`	Symmetric mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Contents#

class evalml.objectives.standard_metrics.AccuracyBinary[source]#

Accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for accuracy score for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for accuracy score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.AccuracyMulticlass[source]#

Accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for accuracy score for multiclass classification.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for accuracy score for multiclass classification.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUC[source]#

AUC score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for AUC score for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for AUC score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.AUCMacro[source]#

AUC score for multiclass classification using macro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for AUC score for multiclass classification using macro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for AUC score for multiclass classification using macro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUCMicro[source]#

AUC score for multiclass classification using micro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for AUC score for multiclass classification using micro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for AUC score for multiclass classification using micro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.AUCWeighted[source]#

AUC Score for multiclass classification using weighted averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.1, 0.2, 0.7],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.2, 0.3]]
>>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for AUC Score for multiclass classification using weighted averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for AUC Score for multiclass classification using weighted averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.BalancedAccuracyBinary[source]#

Balanced accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for accuracy score for balanced accuracy for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for accuracy score for balanced accuracy for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.BalancedAccuracyMulticlass[source]#

Balanced accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for accuracy score for balanced accuracy for multiclass classification.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for accuracy score for balanced accuracy for multiclass classification.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.ExpVariance[source]#

Explained variance score for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	ExpVariance
perfect_score	1.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for explained variance score for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for explained variance score for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1[source]#

F1 score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for F1 score for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for F1 score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.F1Macro[source]#

F1 score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for F1 score for multiclass classification using macro averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for F1 score for multiclass classification using macro averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1Micro[source]#

F1 score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for F1 score for multiclass classification.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for F1 score for multiclass classification.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.F1Weighted[source]#

F1 score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for F1 score for multiclass classification using weighted averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for F1 score for multiclass classification using weighted averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Gini[source]#

Gini coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	Gini
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for Gini coefficient for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for Gini coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.LogLossBinary[source]#

Log Loss for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 19.6601745)

Attributes

expected_range	[0, 1]
greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Binary
perfect_score	0.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for log loss for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for log loss for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.LogLossMulticlass[source]#

Log Loss for multiclass classification.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301)

Attributes

expected_range	[0, 1]
greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Multiclass
perfect_score	0.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for log loss for multiclass classification.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for log loss for multiclass classification.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MAE[source]#

Mean absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for mean absolute error for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for mean absolute error for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MAPE[source]#

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Mean Absolute Percentage Error
perfect_score	0.0
problem_types	[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for mean absolute percentage error for time series regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for mean absolute percentage error for time series regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MASE[source]#

Mean absolute scaled error for time series regression.

Only valid if there exists a nonzero input in y_train. Otherwise, will throw a ValueError.

Example

>>> y_train = pd.Series([5, 0.5, 4, 6, 3, 5, 2])
>>> y_true = pd.Series([3, -0.5, 2, 7, 2])
>>> y_pred = pd.Series([2.5, 0.0, 2, 8, 1.25])
>>> np.testing.assert_almost_equal(MASE().objective_function(y_true, y_pred, y_train), 0.18333333333333335)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Mean Absolute Scaled Error
perfect_score	0.0
problem_types	[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for mean absolute scaled error for time series regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train, X=None, sample_weight=None)[source]#: Objective function for mean absolute scaled error for time series regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MaxError[source]#

Maximum residual error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MaxError
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for maximum residual error for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for maximum residual error for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MCCBinary[source]#

Matthews correlation coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	MCC Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for Matthews correlation coefficient for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for Matthews correlation coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.MCCMulticlass[source]#

Matthews correlation coefficient for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	MCC Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for Matthews correlation coefficient for multiclass classification.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for Matthews correlation coefficient for multiclass classification.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MeanSquaredLogError[source]#

Mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for mean squared log error for regression.
`positive_only`	If True, this objective is only valid for positive data.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for mean squared log error for regression.

positive_only(self)#: If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MedianAE[source]#

Median absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MedianAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for median absolute error for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for median absolute error for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.MSE[source]#

Mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MSE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for mean squared error for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for mean squared error for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Precision[source]#

Precision score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for precision score for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for precision score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.PrecisionMacro[source]#

Precision score for multiclass classification using macro-averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for precision score for multiclass classification using macro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for precision score for multiclass classification using macro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.PrecisionMicro[source]#

Precision score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for precision score for binary classification using micro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for precision score for binary classification using micro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.PrecisionWeighted[source]#

Precision score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for precision score for multiclass classification using weighted averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for precision score for multiclass classification using weighted averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.R2[source]#

Coefficient of determination for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	R2
perfect_score	1
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for coefficient of determination for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for coefficient of determination for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.Recall[source]#

Recall score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for recall score for binary classification.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns: Whether or not an objective can be optimized.
Return type: bool

decision_function(self, ypred_proba, threshold=0.5, X=None)#

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for recall score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#: Validate inputs for scoring.

class evalml.objectives.standard_metrics.RecallMacro[source]#

Recall score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for recall score for multiclass classification using macro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for recall score for multiclass classification using macro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RecallMicro[source]#

Recall score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for recall score for multiclass classification using micro-averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for recall score for multiclass classification using micro-averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RecallWeighted[source]#

Recall score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for recall score for multiclass classification using weighted averaging.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for recall score for multiclass classification using weighted averaging.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RootMeanSquaredError[source]#

Root mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for root mean squared error for regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for root mean squared error for regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.RootMeanSquaredLogError[source]#

Root mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for root mean squared log error for regression.
`positive_only`	If True, this objective is only valid for positive data.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for root mean squared log error for regression.

positive_only(self)#: If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.standard_metrics.SMAPE[source]#

Symmetric mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(SMAPE().objective_function(y_true, y_pred), 18.13652589)

Attributes

expected_range	[0, 200]
greater_is_better	False
is_bounded_like_percentage	True
name	Symmetric Mean Absolute Percentage Error
perfect_score	0.0
problem_types	[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`	Returns whether or not an objective is defined for a problem type.
`objective_function`	Objective function for symmetric mean absolute percentage error for time series regression.
`positive_only`	If True, this objective is only valid for positive data. Defaults to False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)#: Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)[source]#: Objective function for symmetric mean absolute percentage error for time series regression.

positive_only(cls)#: If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
y_train (pd.Series) – Observed training values of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.