Objectives

EvalML standard and custom objectives.

Submodules

• binary_classification_objective
• cost_benefit_matrix
• fraud_cost
• lead_scoring
• multiclass_classification_objective
• objective_base
• regression_objective
• sensitivity_low_alert
• standard_metrics
• time_series_regression_objective
• utils

Package Contents

Classes Summary

AccuracyBinary	Accuracy score for binary classification.
AccuracyMulticlass	Accuracy score for multiclass classification.
AUC	AUC score for binary classification.
AUCMacro	AUC score for multiclass classification using macro averaging.
AUCMicro	AUC score for multiclass classification using micro averaging.
AUCWeighted	AUC Score for multiclass classification using weighted averaging.
BalancedAccuracyBinary	Balanced accuracy score for binary classification.
BalancedAccuracyMulticlass	Balanced accuracy score for multiclass classification.
BinaryClassificationObjective	Base class for all binary classification objectives.
CostBenefitMatrix	Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.
ExpVariance	Explained variance score for regression.
F1	F1 score for binary classification.
F1Macro	F1 score for multiclass classification using macro averaging.
F1Micro	F1 score for multiclass classification using micro averaging.
F1Weighted	F1 score for multiclass classification using weighted averaging.
FraudCost	Score the percentage of money lost of the total transaction amount process due to fraud.
Gini	Gini coefficient for binary classification.
LeadScoring	Lead scoring.
LogLossBinary	Log Loss for binary classification.
LogLossMulticlass	Log Loss for multiclass classification.
MAE	Mean absolute error for regression.
MAPE	Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.
MaxError	Maximum residual error for regression.
MCCBinary	Matthews correlation coefficient for binary classification.
MCCMulticlass	Matthews correlation coefficient for multiclass classification.
MeanSquaredLogError	Mean squared log error for regression.
MedianAE	Median absolute error for regression.
MSE	Mean squared error for regression.
MulticlassClassificationObjective	Base class for all multiclass classification objectives.
ObjectiveBase	Base class for all objectives.
Precision	Precision score for binary classification.
PrecisionMacro	Precision score for multiclass classification using macro-averaging.
PrecisionMicro	Precision score for multiclass classification using micro averaging.
PrecisionWeighted	Precision score for multiclass classification using weighted averaging.
R2	Coefficient of determination for regression.
Recall	Recall score for binary classification.
RecallMacro	Recall score for multiclass classification using macro averaging.
RecallMicro	Recall score for multiclass classification using micro averaging.
RecallWeighted	Recall score for multiclass classification using weighted averaging.
RegressionObjective	Base class for all regression objectives.
RootMeanSquaredError	Root mean squared error for regression.
RootMeanSquaredLogError	Root mean squared log error for regression.
SensitivityLowAlert	Create instance of SensitivityLowAlert.

Functions

get_all_objective_names	Get a list of the names of all objectives.
get_core_objective_names	Get a list of all valid core objectives.
get_core_objectives	Returns all core objective instances associated with the given problem type.
get_default_recommendation_objectives	Get the default recommendation score metrics for the given problem type.
get_non_core_objectives	Get non-core objective classes.
get_objective	Returns the Objective class corresponding to a given objective name.
get_optimization_objectives	Get objectives for optimization.
get_ranking_objectives	Get objectives for pipeline rankings.
normalize_objectives	Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.
organize_objectives	Generate objectives to consider, with optional modifications to the defaults.
ranking_only_objectives	Get ranking-only objective classes.
recommendation_score	Computes a recommendation score for a model given scores for a group of objectives.

Contents

class evalml.objectives.AccuracyBinary

Accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for accuracy score for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for accuracy score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.AccuracyMulticlass

Accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for accuracy score for multiclass classification.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for accuracy score for multiclass classification.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.AUC

AUC score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for AUC score for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for AUC score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.AUCMacro

AUC score for multiclass classification using macro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for AUC score for multiclass classification using macro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for AUC score for multiclass classification using macro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.AUCMicro

AUC score for multiclass classification using micro averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for AUC score for multiclass classification using micro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for AUC score for multiclass classification using micro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.AUCWeighted

AUC Score for multiclass classification using weighted averaging.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.1, 0.0, 0.9],
...           [0.1, 0.3, 0.6],
...           [0.1, 0.2, 0.7],
...           [0.6, 0.1, 0.3],
...           [0.5, 0.2, 0.3]]
>>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	AUC Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for AUC Score for multiclass classification using weighted averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for AUC Score for multiclass classification using weighted averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.BalancedAccuracyBinary

Balanced accuracy score for binary classification.

Example

>>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for accuracy score for balanced accuracy for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for accuracy score for balanced accuracy for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.BalancedAccuracyMulticlass

Balanced accuracy score for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for accuracy score for balanced accuracy for multiclass classification.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for accuracy score for balanced accuracy for multiclass classification.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.BinaryClassificationObjective

Base class for all binary classification objectives.

Attributes

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
expected_range	Returns the expected range of the objective, which is not necessarily the possible ranges.
greater_is_better	Returns a boolean determining if a greater score indicates better model performance.
is_bounded_like_percentage	Returns whether this objective is bounded between 0 and 1, inclusive.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
name	Returns a name describing the objective.
objective_function	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
perfect_score	Returns the score obtained by evaluating this objective on a perfect model.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
score_needs_proba	Returns a boolean determining if the score() method needs probability estimates.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges.

For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates.

This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.CostBenefitMatrix(true_positive, true_negative, false_positive, false_negative)

Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.

Parameters

• true_positive (float) – Cost associated with true positive predictions.

• true_negative (float) – Cost associated with true negative predictions.

• false_positive (float) – Cost associated with false positive predictions.

• false_negative (float) – Cost associated with false negative predictions.

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	Cost Benefit Matrix
perfect_score	None
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Calculates cost-benefit of the using the predicted and true values.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Calculates cost-benefit of the using the predicted and true values.

Parameters

• y_predicted (pd.Series) – Predicted labels.

• y_true (pd.Series) – True labels.

• X (pd.DataFrame) – Ignored.

• sample_weight (pd.DataFrame) – Ignored.

Returns

Cost-benefit matrix score

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.ExpVariance

Explained variance score for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	ExpVariance
perfect_score	1.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for explained variance score for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for explained variance score for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.F1

F1 score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for F1 score for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for F1 score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.F1Macro

F1 score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for F1 score for multiclass classification using macro averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for F1 score for multiclass classification using macro averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.F1Micro

F1 score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for F1 score for multiclass classification.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for F1 score for multiclass classification.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.F1Weighted

F1 score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	F1 Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for F1 score for multiclass classification using weighted averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for F1 score for multiclass classification using weighted averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.FraudCost(retry_percentage=0.5, interchange_fee=0.02, fraud_payout_percentage=1.0, amount_col='amount')

Score the percentage of money lost of the total transaction amount process due to fraud.

Parameters

• retry_percentage (float) – What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to 0.5.

• interchange_fee (float) – How much of each successful transaction you pay. Between 0 and 1. Defaults to 0.02.

• fraud_payout_percentage (float) – Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0.

• amount_col (str) – Name of column in data that contains the amount. Defaults to “amount”.

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	True
name	Fraud Cost
perfect_score	0.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X, sample_weight=None)

Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.

Parameters

• y_predicted (pd.Series) – Predicted fraud labels.

• y_true (pd.Series) – True fraud labels.

• X (pd.DataFrame) – Data with transaction amounts.

• sample_weight (pd.DataFrame) – Ignored.

Returns

Amount lost to fraud per transaction.

Return type

float

Raises

ValueError – If amount_col is not a valid column in the input data.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

evalml.objectives.get_all_objective_names()

Get a list of the names of all objectives.

Returns

Objective names

Return type

list (str)

evalml.objectives.get_core_objective_names()

Get a list of all valid core objectives.

Returns

Objective names.

Return type

list[str]

evalml.objectives.get_core_objectives(problem_type)

Returns all core objective instances associated with the given problem type.

Core objectives are designed to work out-of-the-box for any dataset.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

Examples

>>> for objective in get_core_objectives("regression"):
...     print(objective.name)
ExpVariance
MaxError
MedianAE
MSE
MAE
R2
Root Mean Squared Error
>>> for objective in get_core_objectives("binary"):
...     print(objective.name)
MCC Binary
Log Loss Binary
Gini
AUC
Precision
F1
Balanced Accuracy Binary
Accuracy Binary

evalml.objectives.get_default_recommendation_objectives(problem_type, imbalanced=False)

Get the default recommendation score metrics for the given problem type.

Parameters

• problem_type (str/ProblemType) – Type of problem

• imbalanced (boolean) – For multiclass problems, if the classes are imbalanced. Defaults to False

Returns

Set of string objective names that correspond to ObjectiveBase objectives

evalml.objectives.get_non_core_objectives()

Get non-core objective classes.

Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives before using them in AutoMLSearch.

Returns

List of ObjectiveBase classes

evalml.objectives.get_objective(objective, return_instance=False, **kwargs)

Returns the Objective class corresponding to a given objective name.

Parameters

• objective (str or ObjectiveBase) – Name or instance of the objective class.

• return_instance (bool) – Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments.

• kwargs (Any) – Any keyword arguments to pass into the objective. Only used when return_instance=True.

Returns

ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned.

Raises

• TypeError – If objective is None.

• TypeError – If objective is not a string and not an instance of ObjectiveBase.

• ObjectiveNotFoundError – If input objective is not a valid objective.

• ObjectiveCreationError – If objective cannot be created properly.

evalml.objectives.get_optimization_objectives(problem_type)

Get objectives for optimization.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

evalml.objectives.get_ranking_objectives(problem_type)

Get objectives for pipeline rankings.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

class evalml.objectives.Gini

Gini coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	Gini
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for Gini coefficient for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for Gini coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.LeadScoring(true_positives=1, false_positives=- 1)

Lead scoring.

Parameters

• true_positives (int) – Reward for a true positive. Defaults to 1.

• false_positives (int) – Cost for a false positive. Should be negative. Defaults to -1.

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	Lead Scoring
perfect_score	None
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Calculate the profit per lead.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Calculate the profit per lead.

Parameters

• y_predicted (pd.Series) – Predicted labels

• y_true (pd.Series) – True labels

• X (pd.DataFrame) – Ignored.

• sample_weight (pd.DataFrame) – Ignored.

Returns

Profit per lead

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.LogLossBinary

Log Loss for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 19.6601745)

Attributes

expected_range	[0, 1]
greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Binary
perfect_score	0.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for log loss for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for log loss for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.LogLossMulticlass

Log Loss for multiclass classification.

Example

>>> y_true = [0, 1, 2, 0, 2, 1]
>>> y_pred = [[0.7, 0.2, 0.1],
...           [0.3, 0.5, 0.2],
...           [0.1, 0.3, 0.6],
...           [0.9, 0.1, 0.0],
...           [0.3, 0.1, 0.6],
...           [0.5, 0.5, 0.0]]
>>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301)

Attributes

expected_range	[0, 1]
greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Multiclass
perfect_score	0.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for log loss for multiclass classification.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for log loss for multiclass classification.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MAE

Mean absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for mean absolute error for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for mean absolute error for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MAPE

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Mean Absolute Percentage Error
perfect_score	0.0
problem_types	[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for mean absolute percentage error for time series regression.
positive_only	If True, this objective is only valid for positive data.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for mean absolute percentage error for time series regression.

positive_only(self)

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MaxError

Maximum residual error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MaxError
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for maximum residual error for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for maximum residual error for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MCCBinary

Matthews correlation coefficient for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	MCC Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for Matthews correlation coefficient for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for Matthews correlation coefficient for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.MCCMulticlass

Matthews correlation coefficient for multiclass classification.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	MCC Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for Matthews correlation coefficient for multiclass classification.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for Matthews correlation coefficient for multiclass classification.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MeanSquaredLogError

Mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for mean squared log error for regression.
positive_only	If True, this objective is only valid for positive data.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for mean squared log error for regression.

positive_only(self)

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MedianAE

Median absolute error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MedianAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for median absolute error for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for median absolute error for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MSE

Mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	MSE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for mean squared error for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for mean squared error for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.MulticlassClassificationObjective

Base class for all multiclass classification objectives.

Attributes

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

Methods

calculate_percent_difference	Calculate the percent difference between scores.
expected_range	Returns the expected range of the objective, which is not necessarily the possible ranges.
greater_is_better	Returns a boolean determining if a greater score indicates better model performance.
is_bounded_like_percentage	Returns whether this objective is bounded between 0 and 1, inclusive.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
name	Returns a name describing the objective.
objective_function	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
perfect_score	Returns the score obtained by evaluating this objective on a perfect model.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
score_needs_proba	Returns a boolean determining if the score() method needs probability estimates.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges.

For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates.

This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

evalml.objectives.normalize_objectives(objectives_to_normalize, max_objectives, min_objectives)

Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.

Parameters

• objectives_to_normalize (dict[str,float]) – A dictionary mapping objectives to values

• max_objectives (dict[str,float]) – The mapping of objectives to the maximum values for normalization

• min_objectives (dict[str,float]) – The mapping of objectives to the minimum values for normalization

Returns

A dictionary mapping objective names to their new normalized values

class evalml.objectives.ObjectiveBase

Base class for all objectives.

Attributes

problem_types

None

Methods

calculate_percent_difference	Calculate the percent difference between scores.
expected_range	Returns the expected range of the objective, which is not necessarily the possible ranges.
greater_is_better	Returns a boolean determining if a greater score indicates better model performance.
is_bounded_like_percentage	Returns whether this objective is bounded between 0 and 1, inclusive.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
name	Returns a name describing the objective.
objective_function	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
perfect_score	Returns the score obtained by evaluating this objective on a perfect model.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
score_needs_proba	Returns a boolean determining if the score() method needs probability estimates.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges.

For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates.

This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

evalml.objectives.organize_objectives(problem_type, include=None, exclude=None, imbalanced=False)

Generate objectives to consider, with optional modifications to the defaults.

Parameters

• problem_type (str/ProblemType) – Type of problem

• include (list[str/ObjectiveBase]) – A list of objectives to include beyond the defaults. Defaults to None.

• exclude (list[str/ObjectiveBase]) – A list of objectives to exclude from the defaults. Defaults to None.

• imbalanced (boolean) – For multiclass problems, if the classes are imbalanced. Defaults to False

Returns

List of string objective names that correspond to ObjectiveBase objectives

Raises

• ValueError – If any objectives to include or exclude are not valid for the problem type

• ValueError – If an objective to exclude is not in the default objectives

class evalml.objectives.Precision

Precision score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for precision score for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for precision score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.PrecisionMacro

Precision score for multiclass classification using macro-averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for precision score for multiclass classification using macro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for precision score for multiclass classification using macro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.PrecisionMicro

Precision score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for precision score for binary classification using micro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for precision score for binary classification using micro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.PrecisionWeighted

Precision score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Precision Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for precision score for multiclass classification using weighted averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for precision score for multiclass classification using weighted averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.R2

Coefficient of determination for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036)

Attributes

expected_range	None
greater_is_better	True
is_bounded_like_percentage	False
name	R2
perfect_score	1
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for coefficient of determination for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for coefficient of determination for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

evalml.objectives.ranking_only_objectives()

Get ranking-only objective classes.

Ranking-only objectives are objectives that are useful for evaluating the performance of a model, but should not be used as an optimization objective during AutoMLSearch for various reasons.

Returns

List of ObjectiveBase classes

class evalml.objectives.Recall

Recall score for binary classification.

Example

>>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1])
>>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Apply a learned threshold to predicted probabilities to get predicted classes.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for recall score for binary classification.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

• ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

• threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for recall score for binary classification.

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.

class evalml.objectives.RecallMacro

Recall score for multiclass classification using macro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for recall score for multiclass classification using macro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for recall score for multiclass classification using macro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.RecallMicro

Recall score for multiclass classification using micro averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for recall score for multiclass classification using micro-averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for recall score for multiclass classification using micro-averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.RecallWeighted

Recall score for multiclass classification using weighted averaging.

Example

>>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2])
>>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2])
>>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545)

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Recall Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for recall score for multiclass classification using weighted averaging.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for recall score for multiclass classification using weighted averaging.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

evalml.objectives.recommendation_score(objectives, prioritized_objective=None, custom_weights=None)

Computes a recommendation score for a model given scores for a group of objectives.

This recommendation score is a weighted average of the given objectives, by default all weighted equally. Passing in a prioritized objective will weight that objective with the prioritized weight, and all other objectives will split the remaining weight equally.

Parameters

• objectives (dict[str,float]) – A dictionary mapping objectives to their values. Objectives should be a float between 0 and 1, where higher is better. If the objective does not represent score this way, scores should first be normalized using the normalize_objectives function.

• prioritized_objective (str) – An optional name of a priority objective that should be given heavier weight (50% of the total) than the other objectives contributing to the score. Defaults to None, where all objectives are weighted equally.

• custom_weights (dict[str,float]) – A dictionary mapping objective names to corresponding weights between 0 and 1. If all objectives are listed, should add up to 1. If a subset of objectives are listed, should add up to less than 1, and remaining weight will be evenly distributed between the remaining objectives. Should not be used at the same time as prioritized_objective.

Returns

A value between 0 and 100 representing how strongly we recommend a pipeline given a set of evaluated objectives

Raises

ValueError – If the objective(s) to prioritize are not in the known objectives, or if the custom weight(s) are not a float between 0 and 1.

class evalml.objectives.RegressionObjective

Base class for all regression objectives.

Attributes

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

Methods

calculate_percent_difference	Calculate the percent difference between scores.
expected_range	Returns the expected range of the objective, which is not necessarily the possible ranges.
greater_is_better	Returns a boolean determining if a greater score indicates better model performance.
is_bounded_like_percentage	Returns whether this objective is bounded between 0 and 1, inclusive.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
name	Returns a name describing the objective.
objective_function	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.
perfect_score	Returns the score obtained by evaluating this objective on a perfect model.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
score_needs_proba	Returns a boolean determining if the score() method needs probability estimates.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges.

For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates.

This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.RootMeanSquaredError

Root mean squared error for regression.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for root mean squared error for regression.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for root mean squared error for regression.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.RootMeanSquaredLogError

Root mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError.

Example

>>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2])
>>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75])
>>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204)

Attributes

expected_range	None
greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Objective function for root mean squared log error for regression.
positive_only	If True, this objective is only valid for positive data.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)

Objective function for root mean squared log error for regression.

positive_only(self)

If True, this objective is only valid for positive data.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validates the input based on a few simple checks.

Parameters

• y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples].

• y_true (pd.Series) – Actual class labels of length [n_samples].

Raises

ValueError – If the inputs are malformed.

class evalml.objectives.SensitivityLowAlert(alert_rate=0.01)

Create instance of SensitivityLowAlert.

Parameters

alert_rate (float) – percentage of top scores to classify as high risk.

Attributes

expected_range	[0, 1]
greater_is_better	True
is_bounded_like_percentage	True
name	Sensitivity at Low Alert Rates
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

calculate_percent_difference	Calculate the percent difference between scores.
can_optimize_threshold	Returns a boolean determining if we can optimize the binary classification objective threshold.
decision_function	Determine if an observation is high risk given an alert rate.
is_defined_for_problem_type	Returns whether or not an objective is defined for a problem type.
objective_function	Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.
optimize_threshold	Learn a binary classification threshold which optimizes the current objective.
positive_only	If True, this objective is only valid for positive data. Defaults to False.
score	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
validate_inputs	Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters

• score (float) – A score. Output of the score method of this objective.

• baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, **kwargs)

Determine if an observation is high risk given an alert rate.

Parameters

• ypred_proba (pd.Series) – Predicted probabilities.

• **kwargs – Additional abritrary parameters.

Returns

Whether or not an observation is high risk given an alert rate.

Return type

pd.Series

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, **kwargs)

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.

Parameters

• y_true (pd.Series) – True labels.

• y_predicted (pd.Series) – Predicted labels based on alert_rate.

• **kwargs – Additional abritrary parameters.

Returns

sensitivity using the observations with the top scores as the predicted positive class.

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)

Learn a binary classification threshold which optimizes the current objective.

Parameters

• ypred_proba (pd.Series) – The classifier’s predicted probabilities

• y_true (pd.Series) – The ground truth for the predictions.

• X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

• y_predicted (pd.Series) – Predicted values of length [n_samples]

• y_true (pd.Series) – Actual class labels of length [n_samples]

• X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

• sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)

Validate inputs for scoring.