utils
=================================
.. py:module:: evalml.objectives.utils
.. autoapi-nested-parse::
Utility methods for EvalML objectives.
Module Contents
---------------
Functions
~~~~~~~~~
.. autoapisummary::
:nosignatures:
evalml.objectives.utils.get_all_objective_names
evalml.objectives.utils.get_core_objective_names
evalml.objectives.utils.get_core_objectives
evalml.objectives.utils.get_default_recommendation_objectives
evalml.objectives.utils.get_non_core_objectives
evalml.objectives.utils.get_objective
evalml.objectives.utils.get_optimization_objectives
evalml.objectives.utils.get_ranking_objectives
evalml.objectives.utils.normalize_objectives
evalml.objectives.utils.organize_objectives
evalml.objectives.utils.ranking_only_objectives
evalml.objectives.utils.recommendation_score
Attributes Summary
~~~~~~~~~~~~~~~~~~~
.. autoapisummary::
evalml.objectives.utils.DEFAULT_RECOMMENDATION_OBJECTIVES
Contents
~~~~~~~~~~~~~~~~~~~
.. py:data:: DEFAULT_RECOMMENDATION_OBJECTIVES
.. py:function:: get_all_objective_names()
Get a list of the names of all objectives.
:returns: Objective names
:rtype: list (str)
.. py:function:: get_core_objective_names()
Get a list of all valid core objectives.
:returns: Objective names.
:rtype: list[str]
.. py:function:: get_core_objectives(problem_type)
Returns all core objective instances associated with the given problem type.
Core objectives are designed to work out-of-the-box for any dataset.
:param problem_type: Type of problem
:type problem_type: str/ProblemTypes
:returns: List of ObjectiveBase instances
.. rubric:: Examples
>>> for objective in get_core_objectives("regression"):
... print(objective.name)
ExpVariance
MaxError
MedianAE
MSE
MAE
R2
Root Mean Squared Error
>>> for objective in get_core_objectives("binary"):
... print(objective.name)
MCC Binary
Log Loss Binary
Gini
AUC
Precision
F1
Balanced Accuracy Binary
Accuracy Binary
.. py:function:: get_default_recommendation_objectives(problem_type, imbalanced=False)
Get the default recommendation score metrics for the given problem type.
:param problem_type: Type of problem
:type problem_type: str/ProblemType
:param imbalanced: For multiclass problems, if the classes are imbalanced. Defaults to False
:type imbalanced: boolean
:returns: Set of string objective names that correspond to ObjectiveBase objectives
.. py:function:: get_non_core_objectives()
Get non-core objective classes.
Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives
before using them in AutoMLSearch.
:returns: List of ObjectiveBase classes
.. py:function:: get_objective(objective, return_instance=False, **kwargs)
Returns the Objective class corresponding to a given objective name.
:param objective: Name or instance of the objective class.
:type objective: str or ObjectiveBase
:param return_instance: Whether to return an instance of the objective. This only applies if objective
is of type str. Note that the instance will be initialized with default arguments.
:type return_instance: bool
:param kwargs: Any keyword arguments to pass into the objective. Only used when return_instance=True.
:type kwargs: Any
:returns: ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid
objective name, function will return the class corresponding to that name. If return_instance is True,
an instance of that objective will be returned.
:raises TypeError: If objective is None.
:raises TypeError: If objective is not a string and not an instance of ObjectiveBase.
:raises ObjectiveNotFoundError: If input objective is not a valid objective.
:raises ObjectiveCreationError: If objective cannot be created properly.
.. py:function:: get_optimization_objectives(problem_type)
Get objectives for optimization.
:param problem_type: Type of problem
:type problem_type: str/ProblemTypes
:returns: List of ObjectiveBase instances
.. py:function:: get_ranking_objectives(problem_type)
Get objectives for pipeline rankings.
:param problem_type: Type of problem
:type problem_type: str/ProblemTypes
:returns: List of ObjectiveBase instances
.. py:function:: normalize_objectives(objectives_to_normalize, max_objectives, min_objectives)
Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.
:param objectives_to_normalize: A dictionary mapping objectives to values
:type objectives_to_normalize: dict[str,float]
:param max_objectives: The mapping of objectives to the maximum values for normalization
:type max_objectives: dict[str,float]
:param min_objectives: The mapping of objectives to the minimum values for normalization
:type min_objectives: dict[str,float]
:returns: A dictionary mapping objective names to their new normalized values
.. py:function:: organize_objectives(problem_type, include=None, exclude=None, imbalanced=False)
Generate objectives to consider, with optional modifications to the defaults.
:param problem_type: Type of problem
:type problem_type: str/ProblemType
:param include: A list of objectives to include beyond the defaults. Defaults to None.
:type include: list[str/ObjectiveBase]
:param exclude: A list of objectives to exclude from the defaults. Defaults to None.
:type exclude: list[str/ObjectiveBase]
:param imbalanced: For multiclass problems, if the classes are imbalanced. Defaults to False
:type imbalanced: boolean
:returns: List of string objective names that correspond to ObjectiveBase objectives
:raises ValueError: If any objectives to include or exclude are not valid for the problem type
:raises ValueError: If an objective to exclude is not in the default objectives
.. py:function:: ranking_only_objectives()
Get ranking-only objective classes.
Ranking-only objectives are objectives that are useful for evaluating the performance of a model, but should not
be used as an optimization objective during AutoMLSearch for various reasons.
:returns: List of ObjectiveBase classes
.. py:function:: recommendation_score(objectives, prioritized_objective=None, custom_weights=None)
Computes a recommendation score for a model given scores for a group of objectives.
This recommendation score is a weighted average of the given objectives, by default all weighted equally. Passing
in a prioritized objective will weight that objective with the prioritized weight, and all other objectives will
split the remaining weight equally.
:param objectives: A dictionary mapping objectives to their values. Objectives should be a float between
0 and 1, where higher is better. If the objective does not represent score this way, scores should first be
normalized using the normalize_objectives function.
:type objectives: dict[str,float]
:param prioritized_objective: An optional name of a priority objective that should be given heavier weight (50% of the
total) than the other objectives contributing to the score. Defaults to None, where all objectives are
weighted equally.
:type prioritized_objective: str
:param custom_weights: A dictionary mapping objective names to corresponding weights between 0 and 1.
If all objectives are listed, should add up to 1. If a subset of objectives are listed, should add up to less
than 1, and remaining weight will be evenly distributed between the remaining objectives. Should not be used
at the same time as prioritized_objective.
:type custom_weights: dict[str,float]
:returns: A value between 0 and 100 representing how strongly we recommend a pipeline given a set of evaluated objectives
:raises ValueError: If the objective(s) to prioritize are not in the known objectives, or if the custom weight(s) are not
a float between 0 and 1.