Objectives#
EvalML standard and custom objectives.
Submodules#
Package Contents#
Classes Summary#
Accuracy score for binary classification. 

Accuracy score for multiclass classification. 

AUC score for binary classification. 

AUC score for multiclass classification using macro averaging. 

AUC score for multiclass classification using micro averaging. 

AUC Score for multiclass classification using weighted averaging. 

Balanced accuracy score for binary classification. 

Balanced accuracy score for multiclass classification. 

Base class for all binary classification objectives. 

Score using a costbenefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow. 

Explained variance score for regression. 

F1 score for binary classification. 

F1 score for multiclass classification using macro averaging. 

F1 score for multiclass classification using micro averaging. 

F1 score for multiclass classification using weighted averaging. 

Score the percentage of money lost of the total transaction amount process due to fraud. 

Gini coefficient for binary classification. 

Lead scoring. 

Log Loss for binary classification. 

Log Loss for multiclass classification. 

Mean absolute error for regression. 

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage. 

Maximum residual error for regression. 

Matthews correlation coefficient for binary classification. 

Matthews correlation coefficient for multiclass classification. 

Mean squared log error for regression. 

Median absolute error for regression. 

Mean squared error for regression. 

Base class for all multiclass classification objectives. 

Base class for all objectives. 

Precision score for binary classification. 

Precision score for multiclass classification using macroaveraging. 

Precision score for multiclass classification using micro averaging. 

Precision score for multiclass classification using weighted averaging. 

Coefficient of determination for regression. 

Recall score for binary classification. 

Recall score for multiclass classification using macro averaging. 

Recall score for multiclass classification using micro averaging. 

Recall score for multiclass classification using weighted averaging. 

Base class for all regression objectives. 

Root mean squared error for regression. 

Root mean squared log error for regression. 

Create instance of SensitivityLowAlert. 
Functions#
Get a list of the names of all objectives. 

Get a list of all valid core objectives. 

Returns all core objective instances associated with the given problem type. 

Get noncore objective classes. 

Returns the Objective class corresponding to a given objective name. 

Get objectives for optimization. 

Get objectives for pipeline rankings. 

Get rankingonly objective classes. 
Contents#
 class evalml.objectives.AccuracyBinary[source]#
Accuracy score for binary classification.
Example
>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Accuracy Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for accuracy score for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.AccuracyMulticlass[source]#
Accuracy score for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Accuracy Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for accuracy score for multiclass classification.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.AUC[source]#
AUC score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for AUC score for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.AUCMacro[source]#
AUC score for multiclass classification using macro averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.6, 0.1, 0.3], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for AUC score for multiclass classification using macroaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.AUCMicro[source]#
AUC score for multiclass classification using micro averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC score for multiclass classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for AUC score for multiclass classification using microaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.AUCWeighted[source]#
AUC Score for multiclass classification using weighted averaging.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.1, 0.2, 0.7], ... [0.6, 0.1, 0.3], ... [0.5, 0.2, 0.3]] >>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
AUC Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for AUC Score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for AUC Score for multiclass classification using weighted averaging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.BalancedAccuracyBinary[source]#
Balanced accuracy score for binary classification.
Example
>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Balanced Accuracy Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for balanced accuracy for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for accuracy score for balanced accuracy for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.BalancedAccuracyMulticlass[source]#
Balanced accuracy score for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Balanced Accuracy Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for accuracy score for balanced accuracy for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for accuracy score for balanced accuracy for multiclass classification.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.BinaryClassificationObjective[source]#
Base class for all binary classification objectives.
Attributes
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns the expected range of the objective, which is not necessarily the possible ranges.
Returns a boolean determining if a greater score indicates better model performance.
Returns whether this objective is bounded between 0 and 1, inclusive.
Returns whether or not an objective is defined for a problem type.
Returns a name describing the objective.
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
Learn a binary classification threshold which optimizes the current objective.
Returns the score obtained by evaluating this objective on a perfect model.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Returns a boolean determining if the score() method needs probability estimates.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)[source]#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 property expected_range(cls)#
Returns the expected range of the objective, which is not necessarily the possible ranges.
For example, our expected R2 range is from [1, 1], although the actual range is (inf, 1].
 property greater_is_better(cls)#
Returns a boolean determining if a greater score indicates better model performance.
 property is_bounded_like_percentage(cls)#
Returns whether this objective is bounded between 0 and 1, inclusive.
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 property name(cls)#
Returns a name describing the objective.
 abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)#
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
Numerical value used to calculate score
 optimize_threshold(self, ypred_proba, y_true, X=None)[source]#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 property perfect_score(cls)#
Returns the score obtained by evaluating this objective on a perfect model.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 property score_needs_proba(cls)#
Returns a boolean determining if the score() method needs probability estimates.
This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
 class evalml.objectives.CostBenefitMatrix(true_positive, true_negative, false_positive, false_negative)[source]#
Score using a costbenefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.
 Parameters
true_positive (float) – Cost associated with true positive predictions.
true_negative (float) – Cost associated with true negative predictions.
false_positive (float) – Cost associated with false positive predictions.
false_negative (float) – Cost associated with false negative predictions.
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
Cost Benefit Matrix
perfect_score
None
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Calculates costbenefit of the using the predicted and true values.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Calculates costbenefit of the using the predicted and true values.
 Parameters
y_predicted (pd.Series) – Predicted labels.
y_true (pd.Series) – True labels.
X (pd.DataFrame) – Ignored.
sample_weight (pd.DataFrame) – Ignored.
 Returns
Costbenefit matrix score
 Return type
float
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.ExpVariance[source]#
Explained variance score for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
ExpVariance
perfect_score
1.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for explained variance score for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for explained variance score for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.F1[source]#
F1 score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for F1 score for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.F1Macro[source]#
F1 score for multiclass classification using macro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification using macro averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for F1 score for multiclass classification using macro averaging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.F1Micro[source]#
F1 score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for F1 score for multiclass classification.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.F1Weighted[source]#
F1 score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
F1 Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for F1 score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for F1 score for multiclass classification using weighted averaging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.FraudCost(retry_percentage=0.5, interchange_fee=0.02, fraud_payout_percentage=1.0, amount_col='amount')[source]#
Score the percentage of money lost of the total transaction amount process due to fraud.
 Parameters
retry_percentage (float) – What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to 0.5.
interchange_fee (float) – How much of each successful transaction you pay. Between 0 and 1. Defaults to 0.02.
fraud_payout_percentage (float) – Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0.
amount_col (str) – Name of column in data that contains the amount. Defaults to “amount”.
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
True
name
Fraud Cost
perfect_score
0.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X, sample_weight=None)[source]#
Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.
 Parameters
y_predicted (pd.Series) – Predicted fraud labels.
y_true (pd.Series) – True fraud labels.
X (pd.DataFrame) – Data with transaction amounts.
sample_weight (pd.DataFrame) – Ignored.
 Returns
Amount lost to fraud per transaction.
 Return type
float
 Raises
ValueError – If amount_col is not a valid column in the input data.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 evalml.objectives.get_all_objective_names()[source]#
Get a list of the names of all objectives.
 Returns
Objective names
 Return type
list (str)
 evalml.objectives.get_core_objective_names()[source]#
Get a list of all valid core objectives.
 Returns
Objective names.
 Return type
list[str]
 evalml.objectives.get_core_objectives(problem_type)[source]#
Returns all core objective instances associated with the given problem type.
Core objectives are designed to work outofthebox for any dataset.
 Parameters
problem_type (str/ProblemTypes) – Type of problem
 Returns
List of ObjectiveBase instances
Examples
>>> for objective in get_core_objectives("regression"): ... print(objective.name) ExpVariance MaxError MedianAE MSE MAE R2 Root Mean Squared Error >>> for objective in get_core_objectives("binary"): ... print(objective.name) MCC Binary Log Loss Binary Gini AUC Precision F1 Balanced Accuracy Binary Accuracy Binary
 evalml.objectives.get_non_core_objectives()[source]#
Get noncore objective classes.
Noncore objectives are objectives that are domainspecific. Users typically need to configure these objectives before using them in AutoMLSearch.
 Returns
List of ObjectiveBase classes
 evalml.objectives.get_objective(objective, return_instance=False, **kwargs)[source]#
Returns the Objective class corresponding to a given objective name.
 Parameters
objective (str or ObjectiveBase) – Name or instance of the objective class.
return_instance (bool) – Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments.
kwargs (Any) – Any keyword arguments to pass into the objective. Only used when return_instance=True.
 Returns
ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned.
 Raises
TypeError – If objective is None.
TypeError – If objective is not a string and not an instance of ObjectiveBase.
ObjectiveNotFoundError – If input objective is not a valid objective.
ObjectiveCreationError – If objective cannot be created properly.
 evalml.objectives.get_optimization_objectives(problem_type)[source]#
Get objectives for optimization.
 Parameters
problem_type (str/ProblemTypes) – Type of problem
 Returns
List of ObjectiveBase instances
 evalml.objectives.get_ranking_objectives(problem_type)[source]#
Get objectives for pipeline rankings.
 Parameters
problem_type (str/ProblemTypes) – Type of problem
 Returns
List of ObjectiveBase instances
 class evalml.objectives.Gini[source]#
Gini coefficient for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
Gini
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for Gini coefficient for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for Gini coefficient for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.LeadScoring(true_positives=1, false_positives= 1)[source]#
Lead scoring.
 Parameters
true_positives (int) – Reward for a true positive. Defaults to 1.
false_positives (int) – Cost for a false positive. Should be negative. Defaults to 1.
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
Lead Scoring
perfect_score
None
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Calculate the profit per lead.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Calculate the profit per lead.
 Parameters
y_predicted (pd.Series) – Predicted labels
y_true (pd.Series) – True labels
X (pd.DataFrame) – Ignored.
sample_weight (pd.DataFrame) – Ignored.
 Returns
Profit per lead
 Return type
float
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.LogLossBinary[source]#
Log Loss for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 19.6601745)
Attributes
expected_range
[0, 1]
greater_is_better
False
is_bounded_like_percentage
False
name
Log Loss Binary
perfect_score
0.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for log loss for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for log loss for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.LogLossMulticlass[source]#
Log Loss for multiclass classification.
Example
>>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301)
Attributes
expected_range
[0, 1]
greater_is_better
False
is_bounded_like_percentage
False
name
Log Loss Multiclass
perfect_score
0.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
True
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for log loss for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for log loss for multiclass classification.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MAE[source]#
Mean absolute error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
True
name
MAE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean absolute error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for mean absolute error for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MAPE[source]#
Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.
Only valid for nonzero inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Mean Absolute Percentage Error
perfect_score
0.0
problem_types
[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean absolute percentage error for time series regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for mean absolute percentage error for time series regression.
 positive_only(self)#
If True, this objective is only valid for positive data.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MaxError[source]#
Maximum residual error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MaxError
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for maximum residual error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for maximum residual error for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MCCBinary[source]#
Matthews correlation coefficient for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
MCC Binary
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for Matthews correlation coefficient for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for Matthews correlation coefficient for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.MCCMulticlass[source]#
Matthews correlation coefficient for multiclass classification.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
MCC Multiclass
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for Matthews correlation coefficient for multiclass classification.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for Matthews correlation coefficient for multiclass classification.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MeanSquaredLogError[source]#
Mean squared log error for regression.
Only valid for nonnegative inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Mean Squared Log Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean squared log error for regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for mean squared log error for regression.
 positive_only(self)#
If True, this objective is only valid for positive data.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MedianAE[source]#
Median absolute error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MedianAE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for median absolute error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for median absolute error for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MSE[source]#
Mean squared error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
MSE
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for mean squared error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for mean squared error for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.MulticlassClassificationObjective[source]#
Base class for all multiclass classification objectives.
Attributes
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
Methods
Calculate the percent difference between scores.
Returns the expected range of the objective, which is not necessarily the possible ranges.
Returns a boolean determining if a greater score indicates better model performance.
Returns whether this objective is bounded between 0 and 1, inclusive.
Returns whether or not an objective is defined for a problem type.
Returns a name describing the objective.
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
Returns the score obtained by evaluating this objective on a perfect model.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Returns a boolean determining if the score() method needs probability estimates.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property expected_range(cls)#
Returns the expected range of the objective, which is not necessarily the possible ranges.
For example, our expected R2 range is from [1, 1], although the actual range is (inf, 1].
 property greater_is_better(cls)#
Returns a boolean determining if a greater score indicates better model performance.
 property is_bounded_like_percentage(cls)#
Returns whether this objective is bounded between 0 and 1, inclusive.
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 property name(cls)#
Returns a name describing the objective.
 abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)#
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
Numerical value used to calculate score
 property perfect_score(cls)#
Returns the score obtained by evaluating this objective on a perfect model.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 property score_needs_proba(cls)#
Returns a boolean determining if the score() method needs probability estimates.
This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.ObjectiveBase[source]#
Base class for all objectives.
Attributes
problem_types
None
Methods
Calculate the percent difference between scores.
Returns the expected range of the objective, which is not necessarily the possible ranges.
Returns a boolean determining if a greater score indicates better model performance.
Returns whether this objective is bounded between 0 and 1, inclusive.
Returns whether or not an objective is defined for a problem type.
Returns a name describing the objective.
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
Returns the score obtained by evaluating this objective on a perfect model.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Returns a boolean determining if the score() method needs probability estimates.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)[source]#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property expected_range(cls)#
Returns the expected range of the objective, which is not necessarily the possible ranges.
For example, our expected R2 range is from [1, 1], although the actual range is (inf, 1].
 property greater_is_better(cls)#
Returns a boolean determining if a greater score indicates better model performance.
 property is_bounded_like_percentage(cls)#
Returns whether this objective is bounded between 0 and 1, inclusive.
 classmethod is_defined_for_problem_type(cls, problem_type)[source]#
Returns whether or not an objective is defined for a problem type.
 property name(cls)#
Returns a name describing the objective.
 abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)[source]#
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
Numerical value used to calculate score
 property perfect_score(cls)#
Returns the score obtained by evaluating this objective on a perfect model.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 property score_needs_proba(cls)#
Returns a boolean determining if the score() method needs probability estimates.
This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
 validate_inputs(self, y_true, y_predicted)[source]#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.Precision[source]#
Precision score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for precision score for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.PrecisionMacro[source]#
Precision score for multiclass classification using macroaveraging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for precision score for multiclass classification using macroaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.PrecisionMicro[source]#
Precision score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for binary classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for precision score for binary classification using microaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.PrecisionWeighted[source]#
Precision score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Precision Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for precision score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for precision score for multiclass classification using weighted averaging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.R2[source]#
Coefficient of determination for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036)
Attributes
expected_range
None
greater_is_better
True
is_bounded_like_percentage
False
name
R2
perfect_score
1
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for coefficient of determination for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for coefficient of determination for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 evalml.objectives.ranking_only_objectives()[source]#
Get rankingonly objective classes.
Rankingonly objectives are objectives that are useful for evaluating the performance of a model, but should not be used as an optimization objective during AutoMLSearch for various reasons.
 Returns
List of ObjectiveBase classes
 class evalml.objectives.Recall[source]#
Recall score for binary classification.
Example
>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Apply a learned threshold to predicted probabilities to get predicted classes.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for binary classification.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, threshold=0.5, X=None)#
Apply a learned threshold to predicted probabilities to get predicted classes.
 Parameters
ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
predictions
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for recall score for binary classification.
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.
 class evalml.objectives.RecallMacro[source]#
Recall score for multiclass classification using macro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Macro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using macroaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for recall score for multiclass classification using macroaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.RecallMicro[source]#
Recall score for multiclass classification using micro averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Micro
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using microaveraging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for recall score for multiclass classification using microaveraging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.RecallWeighted[source]#
Recall score for multiclass classification using weighted averaging.
Example
>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545)
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Recall Weighted
perfect_score
1.0
problem_types
[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for recall score for multiclass classification using weighted averaging.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for recall score for multiclass classification using weighted averaging.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.RegressionObjective[source]#
Base class for all regression objectives.
Attributes
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
Methods
Calculate the percent difference between scores.
Returns the expected range of the objective, which is not necessarily the possible ranges.
Returns a boolean determining if a greater score indicates better model performance.
Returns whether this objective is bounded between 0 and 1, inclusive.
Returns whether or not an objective is defined for a problem type.
Returns a name describing the objective.
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
Returns the score obtained by evaluating this objective on a perfect model.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Returns a boolean determining if the score() method needs probability estimates.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property expected_range(cls)#
Returns the expected range of the objective, which is not necessarily the possible ranges.
For example, our expected R2 range is from [1, 1], although the actual range is (inf, 1].
 property greater_is_better(cls)#
Returns a boolean determining if a greater score indicates better model performance.
 property is_bounded_like_percentage(cls)#
Returns whether this objective is bounded between 0 and 1, inclusive.
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 property name(cls)#
Returns a name describing the objective.
 abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)#
Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
Numerical value used to calculate score
 property perfect_score(cls)#
Returns the score obtained by evaluating this objective on a perfect model.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 property score_needs_proba(cls)#
Returns a boolean determining if the score() method needs probability estimates.
This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.RootMeanSquaredError[source]#
Root mean squared error for regression.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Root Mean Squared Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for root mean squared error for regression.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for root mean squared error for regression.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.RootMeanSquaredLogError[source]#
Root mean squared log error for regression.
Only valid for nonnegative inputs. Otherwise, will throw a ValueError.
Example
>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204)
Attributes
expected_range
None
greater_is_better
False
is_bounded_like_percentage
False
name
Root Mean Squared Log Error
perfect_score
0.0
problem_types
[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns whether or not an objective is defined for a problem type.
Objective function for root mean squared log error for regression.
If True, this objective is only valid for positive data.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validates the input based on a few simple checks.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]#
Objective function for root mean squared log error for regression.
 positive_only(self)#
If True, this objective is only valid for positive data.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validates the input based on a few simple checks.
 Parameters
y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].
y_true (pd.Series) – Actual class labels of length [n_samples].
 Raises
ValueError – If the inputs are malformed.
 class evalml.objectives.SensitivityLowAlert(alert_rate=0.01)[source]#
Create instance of SensitivityLowAlert.
 Parameters
alert_rate (float) – percentage of top scores to classify as high risk.
Attributes
expected_range
[0, 1]
greater_is_better
True
is_bounded_like_percentage
True
name
Sensitivity at Low Alert Rates
perfect_score
1.0
problem_types
[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba
False
Methods
Calculate the percent difference between scores.
Returns a boolean determining if we can optimize the binary classification objective threshold.
Determine if an observation is high risk given an alert rate.
Returns whether or not an objective is defined for a problem type.
Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.
Learn a binary classification threshold which optimizes the current objective.
If True, this objective is only valid for positive data. Defaults to False.
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
Validate inputs for scoring.
 classmethod calculate_percent_difference(cls, score, baseline_score)#
Calculate the percent difference between scores.
 Parameters
score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.
 Returns
 The percent difference between the scores. Note that for objectives that can be interpreted
as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.
 Return type
float
 property can_optimize_threshold(cls)#
Returns a boolean determining if we can optimize the binary classification objective threshold.
This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.
 Returns
Whether or not an objective can be optimized.
 Return type
bool
 decision_function(self, ypred_proba, **kwargs)[source]#
Determine if an observation is high risk given an alert rate.
 Parameters
ypred_proba (pd.Series) – Predicted probabilities.
**kwargs – Additional abritrary parameters.
 Returns
Whether or not an observation is high risk given an alert rate.
 Return type
pd.Series
 classmethod is_defined_for_problem_type(cls, problem_type)#
Returns whether or not an objective is defined for a problem type.
 objective_function(self, y_true, y_predicted, **kwargs)[source]#
Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.
 Parameters
y_true (pd.Series) – True labels.
y_predicted (pd.Series) – Predicted labels based on alert_rate.
**kwargs – Additional abritrary parameters.
 Returns
sensitivity using the observations with the top scores as the predicted positive class.
 Return type
float
 optimize_threshold(self, ypred_proba, y_true, X=None)#
Learn a binary classification threshold which optimizes the current objective.
 Parameters
ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.
 Returns
Optimal threshold for this objective.
 Raises
RuntimeError – If objective cannot be optimized.
 positive_only(cls)#
If True, this objective is only valid for positive data. Defaults to False.
 score(self, y_true, y_predicted, X=None, sample_weight=None)#
Returns a numerical score indicating performance based on the differences between the predicted and actual values.
 Parameters
y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result
 Returns
score
 validate_inputs(self, y_true, y_predicted)#
Validate inputs for scoring.