utils#

Utility methods for EvalML objectives.

Module Contents#

Functions#

get_all_objective_names

Get a list of the names of all objectives.

get_core_objective_names

Get a list of all valid core objectives.

get_core_objectives

Returns all core objective instances associated with the given problem type.

get_default_recommendation_objectives

Get the default recommendation score metrics for the given problem type.

get_non_core_objectives

Get non-core objective classes.

get_objective

Returns the Objective class corresponding to a given objective name.

get_optimization_objectives

Get objectives for optimization.

get_ranking_objectives

Get objectives for pipeline rankings.

normalize_objectives

Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.

organize_objectives

Generate objectives to consider, with optional modifications to the defaults.

ranking_only_objectives

Get ranking-only objective classes.

recommendation_score

Computes a recommendation score for a model given scores for a group of objectives.

Contents#

evalml.objectives.utils.DEFAULT_RECOMMENDATION_OBJECTIVES#
evalml.objectives.utils.get_all_objective_names()[source]#

Get a list of the names of all objectives.

Returns

Objective names

Return type

list (str)

evalml.objectives.utils.get_core_objective_names()[source]#

Get a list of all valid core objectives.

Returns

Objective names.

Return type

list[str]

evalml.objectives.utils.get_core_objectives(problem_type)[source]#

Returns all core objective instances associated with the given problem type.

Core objectives are designed to work out-of-the-box for any dataset.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

Examples

>>> for objective in get_core_objectives("regression"):
...     print(objective.name)
ExpVariance
MaxError
MedianAE
MSE
MAE
R2
Root Mean Squared Error
>>> for objective in get_core_objectives("binary"):
...     print(objective.name)
MCC Binary
Log Loss Binary
Gini
AUC
Precision
F1
Balanced Accuracy Binary
Accuracy Binary
evalml.objectives.utils.get_default_recommendation_objectives(problem_type, imbalanced=False)[source]#

Get the default recommendation score metrics for the given problem type.

Parameters
  • problem_type (str/ProblemType) – Type of problem

  • imbalanced (boolean) – For multiclass problems, if the classes are imbalanced. Defaults to False

Returns

Set of string objective names that correspond to ObjectiveBase objectives

evalml.objectives.utils.get_non_core_objectives()[source]#

Get non-core objective classes.

Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives before using them in AutoMLSearch.

Returns

List of ObjectiveBase classes

evalml.objectives.utils.get_objective(objective, return_instance=False, **kwargs)[source]#

Returns the Objective class corresponding to a given objective name.

Parameters
  • objective (str or ObjectiveBase) – Name or instance of the objective class.

  • return_instance (bool) – Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments.

  • kwargs (Any) – Any keyword arguments to pass into the objective. Only used when return_instance=True.

Returns

ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned.

Raises
  • TypeError – If objective is None.

  • TypeError – If objective is not a string and not an instance of ObjectiveBase.

  • ObjectiveNotFoundError – If input objective is not a valid objective.

  • ObjectiveCreationError – If objective cannot be created properly.

evalml.objectives.utils.get_optimization_objectives(problem_type)[source]#

Get objectives for optimization.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

evalml.objectives.utils.get_ranking_objectives(problem_type)[source]#

Get objectives for pipeline rankings.

Parameters

problem_type (str/ProblemTypes) – Type of problem

Returns

List of ObjectiveBase instances

evalml.objectives.utils.normalize_objectives(objectives_to_normalize, max_objectives, min_objectives)[source]#

Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.

Parameters
  • objectives_to_normalize (dict[str,float]) – A dictionary mapping objectives to values

  • max_objectives (dict[str,float]) – The mapping of objectives to the maximum values for normalization

  • min_objectives (dict[str,float]) – The mapping of objectives to the minimum values for normalization

Returns

A dictionary mapping objective names to their new normalized values

evalml.objectives.utils.organize_objectives(problem_type, include=None, exclude=None, imbalanced=False)[source]#

Generate objectives to consider, with optional modifications to the defaults.

Parameters
  • problem_type (str/ProblemType) – Type of problem

  • include (list[str/ObjectiveBase]) – A list of objectives to include beyond the defaults. Defaults to None.

  • exclude (list[str/ObjectiveBase]) – A list of objectives to exclude from the defaults. Defaults to None.

  • imbalanced (boolean) – For multiclass problems, if the classes are imbalanced. Defaults to False

Returns

List of string objective names that correspond to ObjectiveBase objectives

Raises
  • ValueError – If any objectives to include or exclude are not valid for the problem type

  • ValueError – If an objective to exclude is not in the default objectives

evalml.objectives.utils.ranking_only_objectives()[source]#

Get ranking-only objective classes.

Ranking-only objectives are objectives that are useful for evaluating the performance of a model, but should not be used as an optimization objective during AutoMLSearch for various reasons.

Returns

List of ObjectiveBase classes

evalml.objectives.utils.recommendation_score(objectives, prioritized_objective=None, custom_weights=None)[source]#

Computes a recommendation score for a model given scores for a group of objectives.

This recommendation score is a weighted average of the given objectives, by default all weighted equally. Passing in a prioritized objective will weight that objective with the prioritized weight, and all other objectives will split the remaining weight equally.

Parameters
  • objectives (dict[str,float]) – A dictionary mapping objectives to their values. Objectives should be a float between 0 and 1, where higher is better. If the objective does not represent score this way, scores should first be normalized using the normalize_objectives function.

  • prioritized_objective (str) – An optional name of a priority objective that should be given heavier weight (50% of the total) than the other objectives contributing to the score. Defaults to None, where all objectives are weighted equally.

  • custom_weights (dict[str,float]) – A dictionary mapping objective names to corresponding weights between 0 and 1. If all objectives are listed, should add up to 1. If a subset of objectives are listed, should add up to less than 1, and remaining weight will be evenly distributed between the remaining objectives. Should not be used at the same time as prioritized_objective.

Returns

A value between 0 and 100 representing how strongly we recommend a pipeline given a set of evaluated objectives

Raises

ValueError – If the objective(s) to prioritize are not in the known objectives, or if the custom weight(s) are not a float between 0 and 1.