Objectives¶

Submodules¶

Package Contents¶

Classes Summary¶

`AccuracyBinary`	Accuracy score for binary classification.
`AccuracyMulticlass`	Accuracy score for multiclass classification.
`AUC`	AUC score for binary classification.
`AUCMacro`	AUC score for multiclass classification using macro averaging.
`AUCMicro`	AUC score for multiclass classification using micro averaging.
`AUCWeighted`	AUC Score for multiclass classification using weighted averaging.
`BalancedAccuracyBinary`	Balanced accuracy score for binary classification.
`BalancedAccuracyMulticlass`	Balanced accuracy score for multiclass classification.
`BinaryClassificationObjective`	Base class for all binary classification objectives.
`CostBenefitMatrix`	Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric
`ExpVariance`	Explained variance score for regression.
`F1`	F1 score for binary classification.
`F1Macro`	F1 score for multiclass classification using macro averaging.
`F1Micro`	F1 score for multiclass classification using micro averaging.
`F1Weighted`	F1 score for multiclass classification using weighted averaging.
`FraudCost`	Score the percentage of money lost of the total transaction amount process due to fraud.
`LeadScoring`	Lead scoring.
`LogLossBinary`	Log Loss for binary classification.
`LogLossMulticlass`	Log Loss for multiclass classification.
`MAE`	Mean absolute error for regression.
`MAPE`	Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.
`MaxError`	Maximum residual error for regression.
`MCCBinary`	Matthews correlation coefficient for binary classification.
`MCCMulticlass`	Matthews correlation coefficient for multiclass classification.
`MeanSquaredLogError`	Mean squared log error for regression.
`MedianAE`	Median absolute error for regression.
`MSE`	Mean squared error for regression.
`MulticlassClassificationObjective`	Base class for all multiclass classification objectives.
`ObjectiveBase`	Base class for all objectives.
`Precision`	Precision score for binary classification.
`PrecisionMacro`	Precision score for multiclass classification using macro averaging.
`PrecisionMicro`	Precision score for multiclass classification using micro averaging.
`PrecisionWeighted`	Precision score for multiclass classification using weighted averaging.
`R2`	Coefficient of determination for regression.
`Recall`	Recall score for binary classification.
`RecallMacro`	Recall score for multiclass classification using macro averaging.
`RecallMicro`	Recall score for multiclass classification using micro averaging.
`RecallWeighted`	Recall score for multiclass classification using weighted averaging.
`RegressionObjective`	Base class for all regression objectives.
`RootMeanSquaredError`	Root mean squared error for regression.
`RootMeanSquaredLogError`	Root mean squared log error for regression.
`SensitivityLowAlert`	Base class for all binary classification objectives.

Functions¶

`get_all_objective_names`	Get a list of the names of all objectives.
`get_core_objective_names`	Get a list of all valid core objectives.
`get_core_objectives`	Returns all core objective instances associated with the given problem type.
`get_non_core_objectives`	Get non-core objective classes.
`get_objective`	Returns the Objective class corresponding to a given objective name.

Contents¶

class evalml.objectives.AccuracyBinary[source]¶

Accuracy score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AccuracyMulticlass[source]¶

Accuracy score for multiclass classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUC[source]¶

AUC score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	AUC
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCMacro[source]¶

AUC score for multiclass classification using macro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	AUC Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCMicro[source]¶

AUC score for multiclass classification using micro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	AUC Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.AUCWeighted[source]¶

AUC Score for multiclass classification using weighted averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	AUC Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BalancedAccuracyBinary[source]¶

Balanced accuracy score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BalancedAccuracyMulticlass[source]¶

Balanced accuracy score for multiclass classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Balanced Accuracy Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.BinaryClassificationObjective[source]¶

Base class for all binary classification objectives.

Attributes

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`greater_is_better`	Returns a boolean determining if a greater score indicates better model performance.
`is_bounded_like_percentage`	Returns whether this objective is bounded between 0 and 1, inclusive.
`is_defined_for_problem_type`
`name`	Returns a name describing the objective.
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`perfect_score`	Returns the score obtained by evaluating this objective on a perfect model.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`score_needs_proba`	Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)[source]¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

property greater_is_better(cls)¶: Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)¶: Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)¶

property name(cls)¶: Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)[source]¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

property perfect_score(cls)¶: Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)¶: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)[source]¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.CostBenefitMatrix(true_positive, true_negative, false_positive, false_negative)[source]¶

Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.

Parameters

true_positive (float) – Cost associated with true positive predictions
true_negative (float) – Cost associated with true negative predictions
false_positive (float) – Cost associated with false positive predictions
false_negative (float) – Cost associated with false negative predictions

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	Cost Benefit Matrix
perfect_score	None
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Calculates cost-benefit of the using the predicted and true values.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Calculates cost-benefit of the using the predicted and true values.

Parameters

y_predicted (pd.Series) – Predicted labels
y_true (pd.Series) – True labels
X (pd.DataFrame) – Ignored.
sample_weight (pd.DataFrame) – Ignored.

Returns

Cost-benefit matrix score

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.ExpVariance[source]¶

Explained variance score for regression.

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	ExpVariance
perfect_score	1.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1[source]¶

F1 score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	F1
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Macro[source]¶

F1 score for multiclass classification using macro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	F1 Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Micro[source]¶

F1 score for multiclass classification using micro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	F1 Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.F1Weighted[source]¶

F1 score for multiclass classification using weighted averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	F1 Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.FraudCost(retry_percentage=0.5, interchange_fee=0.02, fraud_payout_percentage=1.0, amount_col='amount')[source]¶

Score the percentage of money lost of the total transaction amount process due to fraud.

Parameters

retry_percentage (float) – What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to 0.5.
interchange_fee (float) – How much of each successful transaction you pay. Between 0 and 1. Defaults to 0.02.
fraud_payout_percentage (float) – Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0.
amount_col (str) – Name of column in data that contains the amount. Defaults to “amount”.

Attributes

greater_is_better	False
is_bounded_like_percentage	True
name	Fraud Cost
perfect_score	0.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X, sample_weight=None)[source]¶

Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount.

Parameters

y_predicted (pd.Series) – Predicted fraud labels
y_true (pd.Series) – True fraud labels
X (pd.DataFrame) – Data with transaction amounts
sample_weight (pd.DataFrame) – Ignored.

Returns

Amount lost to fraud per transaction

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

evalml.objectives.get_all_objective_names()[source]¶

Get a list of the names of all objectives.

Returns: Objective names
Return type: list (str)

evalml.objectives.get_core_objective_names()[source]¶

Get a list of all valid core objectives.

Returns: Objective names.
Return type: list[str]

evalml.objectives.get_core_objectives(problem_type)[source]¶

Returns all core objective instances associated with the given problem type.

Core objectives are designed to work out-of-the-box for any dataset.

Parameters: problem_type (str/ProblemTypes) – Type of problem
Returns: List of ObjectiveBase instances

evalml.objectives.get_non_core_objectives()[source]¶

Get non-core objective classes.

Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives before using them in AutoMLSearch.

Returns: List of ObjectiveBase classes

evalml.objectives.get_objective(objective, return_instance=False, **kwargs)[source]¶

Returns the Objective class corresponding to a given objective name.

Parameters

objective (str or ObjectiveBase) – Name or instance of the objective class.
return_instance (bool) – Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments.
kwargs (Any) – Any keyword arguments to pass into the objective. Only used when return_instance=True.

Returns

ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned.

class evalml.objectives.LeadScoring(true_positives=1, false_positives=- 1)[source]¶

Lead scoring.

Parameters

true_positives (int) – Reward for a true positive. Defaults to 1.
false_positives (int) – Cost for a false positive. Should be negative. Defaults to -1.

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	Lead Scoring
perfect_score	None
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Calculate the profit per lead.
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Calculate the profit per lead.

Parameters

y_predicted (pd.Series) – Predicted labels
y_true (pd.Series) – True labels
X (pd.DataFrame) – Ignored.
sample_weight (pd.DataFrame) – Ignored.

Returns

Profit per lead

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.LogLossBinary[source]¶

Log Loss for binary classification.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Binary
perfect_score	0.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.LogLossMulticlass[source]¶

Log Loss for multiclass classification.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Log Loss Multiclass
perfect_score	0.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	True

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MAE[source]¶

Mean absolute error for regression.

Attributes

greater_is_better	False
is_bounded_like_percentage	True
name	MAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MAPE[source]¶

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

Only valid for nonzero inputs. Otherwise, will throw a ValueError

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Mean Absolute Percentage Error
perfect_score	0.0
problem_types	[ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(self)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MaxError[source]¶

Maximum residual error for regression.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	MaxError
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MCCBinary[source]¶

Matthews correlation coefficient for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	MCC Binary
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MCCMulticlass[source]¶

Matthews correlation coefficient for multiclass classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	MCC Multiclass
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MeanSquaredLogError[source]¶

Mean squared log error for regression.

Only valid for nonnegative inputs. Otherwise, will throw a ValueError

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(self)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MedianAE[source]¶

Median absolute error for regression.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	MedianAE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MSE[source]¶

Mean squared error for regression.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	MSE
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.MulticlassClassificationObjective[source]¶

Base class for all multiclass classification objectives.

Attributes

problem_types

[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`greater_is_better`	Returns a boolean determining if a greater score indicates better model performance.
`is_bounded_like_percentage`	Returns whether this objective is bounded between 0 and 1, inclusive.
`is_defined_for_problem_type`
`name`	Returns a name describing the objective.
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`perfect_score`	Returns the score obtained by evaluating this objective on a perfect model.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`score_needs_proba`	Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property greater_is_better(cls)¶: Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)¶: Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)¶

property name(cls)¶: Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

property perfect_score(cls)¶: Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)¶: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.ObjectiveBase[source]¶

Base class for all objectives.

Attributes

problem_types

None

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`greater_is_better`	Returns a boolean determining if a greater score indicates better model performance.
`is_bounded_like_percentage`	Returns whether this objective is bounded between 0 and 1, inclusive.
`is_defined_for_problem_type`
`name`	Returns a name describing the objective.
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`perfect_score`	Returns the score obtained by evaluating this objective on a perfect model.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`score_needs_proba`	Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)[source]¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property greater_is_better(cls)¶: Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)¶: Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)[source]¶

property name(cls)¶: Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

property perfect_score(cls)¶: Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)¶: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)[source]¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.Precision[source]¶

Precision score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Precision
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionMacro[source]¶

Precision score for multiclass classification using macro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Precision Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionMicro[source]¶

Precision score for multiclass classification using micro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Precision Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.PrecisionWeighted[source]¶

Precision score for multiclass classification using weighted averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Precision Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.R2[source]¶

Coefficient of determination for regression.

Attributes

greater_is_better	True
is_bounded_like_percentage	False
name	R2
perfect_score	1
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.Recall[source]¶

Recall score for binary classification.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Recall
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Apply a learned threshold to predicted probabilities to get predicted classes.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, threshold=0.5, X=None)¶

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters

ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities
threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallMacro[source]¶

Recall score for multiclass classification using macro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Recall Macro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallMicro[source]¶

Recall score for multiclass classification using micro averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Recall Micro
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RecallWeighted[source]¶

Recall score for multiclass classification using weighted averaging.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Recall Weighted
perfect_score	1.0
problem_types	[ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RegressionObjective[source]¶

Base class for all regression objectives.

Attributes

problem_types

[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`greater_is_better`	Returns a boolean determining if a greater score indicates better model performance.
`is_bounded_like_percentage`	Returns whether this objective is bounded between 0 and 1, inclusive.
`is_defined_for_problem_type`
`name`	Returns a name describing the objective.
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`perfect_score`	Returns the score obtained by evaluating this objective on a perfect model.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`score_needs_proba`	Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property greater_is_better(cls)¶: Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)¶: Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)¶

property name(cls)¶: Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

property perfect_score(cls)¶: Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)¶: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RootMeanSquaredError[source]¶

Root mean squared error for regression.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.RootMeanSquaredLogError[source]¶

Root mean squared log error for regression.

Only valid for nonnegative inputs.Otherwise, will throw a ValueError.

Attributes

greater_is_better	False
is_bounded_like_percentage	False
name	Root Mean Squared Log Error
perfect_score	0.0
problem_types	[ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`is_defined_for_problem_type`
`objective_function`	Computes the relative value of the provided predictions compared to the actual labels, according a specified metric
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, X=None, sample_weight=None)[source]¶

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric

Returns: Numerical value used to calculate score

positive_only(self)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

class evalml.objectives.SensitivityLowAlert(alert_rate=0.01)[source]¶

Base class for all binary classification objectives.

Attributes

greater_is_better	True
is_bounded_like_percentage	True
name	Sensitivity at Low Alert Rates
perfect_score	1.0
problem_types	[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]
score_needs_proba	False

Methods

`calculate_percent_difference`	Calculate the percent difference between scores.
`can_optimize_threshold`	Returns a boolean determining if we can optimize the binary classification objective threshold.
`decision_function`	Determine if an observation is high risk given an alert rate
`is_defined_for_problem_type`
`objective_function`	Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class
`optimize_threshold`	Learn a binary classification threshold which optimizes the current objective.
`positive_only`	If True, this objective is only valid for positive data. Default False.
`score`	Returns a numerical score indicating performance based on the differences between the predicted and actual values.
`validate_inputs`	Validates the input based on a few simple checks.

classmethod calculate_percent_difference(cls, score, baseline_score)¶

Calculate the percent difference between scores.

Parameters

score (float) – A score. Output of the score method of this objective.
baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted: as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)¶: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

decision_function(self, ypred_proba, **kwargs)[source]¶

Determine if an observation is high risk given an alert rate

Parameters: ypred_proba (pd.Series) – Predicted probabilities

classmethod is_defined_for_problem_type(cls, problem_type)¶

objective_function(self, y_true, y_predicted, **kwargs)[source]¶

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class

Parameters

y_true (pd.Series) – True labels
y_predicted (pd.Series) – Predicted labels based on alert_rate

Returns

sensitivity using the observations with the top scores as the predicted positive class

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)¶

Learn a binary classification threshold which optimizes the current objective.

Parameters

ypred_proba (pd.Series) – The classifier’s predicted probabilities
y_true (pd.Series) – The ground truth for the predictions.
X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective

positive_only(cls)¶: If True, this objective is only valid for positive data. Default False.

score(self, y_true, y_predicted, X=None, sample_weight=None)¶

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters

y_predicted (pd.Series) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]
X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score
sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)¶

Validates the input based on a few simple checks.

Parameters

y_predicted (pd.Series, or pd.DataFrame) – Predicted values of length [n_samples]
y_true (pd.Series) – Actual class labels of length [n_samples]

Returns

None

permutation_importance binary_classification_objective