standard_metrics¶
Standard machine learning objective functions.
Module Contents¶
Classes Summary¶
Accuracy score for binary classification. 

Accuracy score for multiclass classification. 

AUC score for binary classification. 

AUC score for multiclass classification using macro averaging. 

AUC score for multiclass classification using micro averaging. 

AUC Score for multiclass classification using weighted averaging. 

Balanced accuracy score for binary classification. 

Balanced accuracy score for multiclass classification. 

Explained variance score for regression. 

F1 score for binary classification. 

F1 score for multiclass classification using macro averaging. 

F1 score for multiclass classification using micro averaging. 

F1 score for multiclass classification using weighted averaging. 

Gini coefficient for binary classification. 

Log Loss for binary classification. 

Log Loss for multiclass classification. 

Mean absolute error for regression. 

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage. 

Maximum residual error for regression. 

Matthews correlation coefficient for binary classification. 

Matthews correlation coefficient for multiclass classification. 

Mean squared log error for regression. 

Median absolute error for regression. 

Mean squared error for regression. 

Precision score for binary classification. 

Precision score for multiclass classification using macroaveraging. 

Precision score for multiclass classification using micro averaging. 

Precision score for multiclass classification using weighted averaging. 

Coefficient of determination for regression. 

Recall score for binary classification. 

Recall score for multiclass classification using macro averaging. 

Recall score for multiclass classification using micro averaging. 

Recall score for multiclass classification using weighted averaging. 

Root mean squared error for regression. 

Root mean squared log error for regression. 
Contents¶

class
evalml.objectives.standard_metrics.
AccuracyBinary
[source]¶ Accuracy score for binary classification.
Example
>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Accuracy Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for accuracy score for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
AccuracyMulticlass
[source]¶ Accuracy score for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Accuracy Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for accuracy score for multiclass classification.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
AUC
[source]¶ AUC score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for AUC score for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
AUCMacro
[source]¶ AUC score for multiclass classification using macro averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.6, 0.1, 0.3], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for AUC score for multiclass classification using macroaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
AUCMicro
[source]¶ AUC score for multiclass classification using micro averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for multiclass classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for AUC score for multiclass classification using microaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
AUCWeighted
[source]¶ AUC Score for multiclass classification using weighted averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.1, 0.2, 0.7], ... [0.6, 0.1, 0.3], ... [0.5, 0.2, 0.3]] >>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC Score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for AUC Score for multiclass classification using weighted averaging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
BalancedAccuracyBinary
[source]¶ Balanced accuracy score for binary classification.
Example
>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Balanced Accuracy Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for balanced accuracy for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for accuracy score for balanced accuracy for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
BalancedAccuracyMulticlass
[source]¶ Balanced accuracy score for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Balanced Accuracy Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for balanced accuracy for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for accuracy score for balanced accuracy for multiclass classification.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
ExpVariance
[source]¶ Explained variance score for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
ExpVariance
perfect_score
1.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for explained variance score for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for explained variance score for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
F1
[source]¶ F1 score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for F1 score for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
F1Macro
[source]¶ F1 score for multiclass classification using macro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification using macro averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for F1 score for multiclass classification using macro averaging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
F1Micro
[source]¶ F1 score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for F1 score for multiclass classification.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
F1Weighted
[source]¶ F1 score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for F1 score for multiclass classification using weighted averaging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
Gini
[source]¶ Gini coefficient for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
Gini
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for Gini coefficient for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for Gini coefficient for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
LogLossBinary
[source]¶ Log Loss for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 18.8393325)
Attributes
expected_range
[0, 1]
greater_is_better
False
is_bounded_like_percentage
False
name
Log Loss Binary
perfect_score
0.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for log loss for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for log loss for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
LogLossMulticlass
[source]¶ Log Loss for multiclass classification.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301)
Attributes
expected_range
[0, 1]
greater_is_better
False
is_bounded_like_percentage
False
name
Log Loss Multiclass
perfect_score
0.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for log loss for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for log loss for multiclass classification.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MAE
[source]¶ Mean absolute error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
True
name
MAE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean absolute error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for mean absolute error for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MAPE
[source]¶ Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.
Only valid for nonzero inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Mean Absolute Percentage Error
perfect_score
0.0
problem_types
[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean absolute percentage error for time series regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for mean absolute percentage error for time series regression.

positive_only
(self)¶ If True, this objective is only valid for positive data.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MaxError
[source]¶ Maximum residual error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MaxError
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for maximum residual error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for maximum residual error for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MCCBinary
[source]¶ Matthews correlation coefficient for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
MCC Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for Matthews correlation coefficient for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for Matthews correlation coefficient for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
MCCMulticlass
[source]¶ Matthews correlation coefficient for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
MCC Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for Matthews correlation coefficient for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for Matthews correlation coefficient for multiclass classification.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MeanSquaredLogError
[source]¶ Mean squared log error for regression.
Only valid for nonnegative inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Mean Squared Log Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean squared log error for regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for mean squared log error for regression.

positive_only
(self)¶ If True, this objective is only valid for positive data.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MedianAE
[source]¶ Median absolute error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MedianAE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for median absolute error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for median absolute error for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
MSE
[source]¶ Mean squared error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MSE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean squared error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for mean squared error for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
Precision
[source]¶ Precision score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for precision score for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
PrecisionMacro
[source]¶ Precision score for multiclass classification using macroaveraging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for precision score for multiclass classification using macroaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
PrecisionMicro
[source]¶ Precision score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for binary classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for precision score for binary classification using microaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
PrecisionWeighted
[source]¶ Precision score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for precision score for multiclass classification using weighted averaging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
R2
[source]¶ Coefficient of determination for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
R2
perfect_score
1
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for coefficient of determination for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for coefficient of determination for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
Recall
[source]¶ Recall score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

property
can_optimize_threshold
(cls)¶ Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool

decision_function
(self, ypred_proba, threshold=0.5, X=None)¶ Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for recall score for binary classification.

optimize_threshold
(self, ypred_proba, y_true, X=None)¶ Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validate inputs for scoring.

classmethod

class
evalml.objectives.standard_metrics.
RecallMacro
[source]¶ Recall score for multiclass classification using macro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for recall score for multiclass classification using macroaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
RecallMicro
[source]¶ Recall score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for recall score for multiclass classification using microaveraging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
RecallWeighted
[source]¶ Recall score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for recall score for multiclass classification using weighted averaging.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
RootMeanSquaredError
[source]¶ Root mean squared error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Root Mean Squared Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for root mean squared error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for root mean squared error for regression.

positive_only
(cls)¶ If True, this objective is only valid for positive data. Defaults to False.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod

class
evalml.objectives.standard_metrics.
RootMeanSquaredLogError
[source]¶ Root mean squared log error for regression.
Only valid for nonnegative inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Root Mean Squared Log Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for root mean squared log error for regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.

classmethod
calculate_percent_difference
(cls, score, baseline_score)¶ Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float

classmethod
is_defined_for_problem_type
(cls, problem_type)¶ Returns whether or not an objective is defined for a problem type.

objective_function
(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶ Objective function for root mean squared log error for regression.

positive_only
(self)¶ If True, this objective is only valid for positive data.

score
(self, y_true, y_predicted, X=None, sample_weight=None)¶ Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score

validate_inputs
(self, y_true, y_predicted)¶ Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.

classmethod