Objectives =========================== .. py:module:: evalml.objectives .. autoapi-nested-parse:: EvalML standard and custom objectives. Submodules ---------- .. toctree:: :titlesonly: :maxdepth: 1 binary_classification_objective/index.rst cost_benefit_matrix/index.rst fraud_cost/index.rst lead_scoring/index.rst multiclass_classification_objective/index.rst objective_base/index.rst regression_objective/index.rst sensitivity_low_alert/index.rst standard_metrics/index.rst time_series_regression_objective/index.rst utils/index.rst Package Contents ---------------- Classes Summary ~~~~~~~~~~~~~~~ .. autoapisummary:: evalml.objectives.AccuracyBinary evalml.objectives.AccuracyMulticlass evalml.objectives.AUC evalml.objectives.AUCMacro evalml.objectives.AUCMicro evalml.objectives.AUCWeighted evalml.objectives.BalancedAccuracyBinary evalml.objectives.BalancedAccuracyMulticlass evalml.objectives.BinaryClassificationObjective evalml.objectives.CostBenefitMatrix evalml.objectives.ExpVariance evalml.objectives.F1 evalml.objectives.F1Macro evalml.objectives.F1Micro evalml.objectives.F1Weighted evalml.objectives.FraudCost evalml.objectives.Gini evalml.objectives.LeadScoring evalml.objectives.LogLossBinary evalml.objectives.LogLossMulticlass evalml.objectives.MAE evalml.objectives.MAPE evalml.objectives.MASE evalml.objectives.MaxError evalml.objectives.MCCBinary evalml.objectives.MCCMulticlass evalml.objectives.MeanSquaredLogError evalml.objectives.MedianAE evalml.objectives.MSE evalml.objectives.MulticlassClassificationObjective evalml.objectives.ObjectiveBase evalml.objectives.Precision evalml.objectives.PrecisionMacro evalml.objectives.PrecisionMicro evalml.objectives.PrecisionWeighted evalml.objectives.R2 evalml.objectives.Recall evalml.objectives.RecallMacro evalml.objectives.RecallMicro evalml.objectives.RecallWeighted evalml.objectives.RegressionObjective evalml.objectives.RootMeanSquaredError evalml.objectives.RootMeanSquaredLogError evalml.objectives.SensitivityLowAlert evalml.objectives.SMAPE Functions ~~~~~~~~~ .. autoapisummary:: :nosignatures: evalml.objectives.get_all_objective_names evalml.objectives.get_core_objective_names evalml.objectives.get_core_objectives evalml.objectives.get_default_recommendation_objectives evalml.objectives.get_non_core_objectives evalml.objectives.get_objective evalml.objectives.get_optimization_objectives evalml.objectives.get_ranking_objectives evalml.objectives.normalize_objectives evalml.objectives.organize_objectives evalml.objectives.ranking_only_objectives evalml.objectives.recommendation_score Contents ~~~~~~~~~~~~~~~~~~~ .. py:class:: AccuracyBinary Accuracy score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AccuracyBinary().objective_function(y_true, y_pred), 0.6363636) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Accuracy Binary * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AccuracyBinary.calculate_percent_difference evalml.objectives.AccuracyBinary.can_optimize_threshold evalml.objectives.AccuracyBinary.decision_function evalml.objectives.AccuracyBinary.is_defined_for_problem_type evalml.objectives.AccuracyBinary.objective_function evalml.objectives.AccuracyBinary.optimize_threshold evalml.objectives.AccuracyBinary.positive_only evalml.objectives.AccuracyBinary.score evalml.objectives.AccuracyBinary.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for accuracy score for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: AccuracyMulticlass Accuracy score for multiclass classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(AccuracyMulticlass().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Accuracy Multiclass * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AccuracyMulticlass.calculate_percent_difference evalml.objectives.AccuracyMulticlass.is_defined_for_problem_type evalml.objectives.AccuracyMulticlass.objective_function evalml.objectives.AccuracyMulticlass.positive_only evalml.objectives.AccuracyMulticlass.score evalml.objectives.AccuracyMulticlass.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for accuracy score for multiclass classification. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: AUC AUC score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(AUC().objective_function(y_true, y_pred), 0.5714285) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - AUC * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AUC.calculate_percent_difference evalml.objectives.AUC.can_optimize_threshold evalml.objectives.AUC.decision_function evalml.objectives.AUC.is_defined_for_problem_type evalml.objectives.AUC.objective_function evalml.objectives.AUC.optimize_threshold evalml.objectives.AUC.positive_only evalml.objectives.AUC.score evalml.objectives.AUC.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for AUC score for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: AUCMacro AUC score for multiclass classification using macro averaging. .. rubric:: Example >>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.6, 0.1, 0.3], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMacro().objective_function(y_true, y_pred), 0.75) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - AUC Macro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AUCMacro.calculate_percent_difference evalml.objectives.AUCMacro.is_defined_for_problem_type evalml.objectives.AUCMacro.objective_function evalml.objectives.AUCMacro.positive_only evalml.objectives.AUCMacro.score evalml.objectives.AUCMacro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for AUC score for multiclass classification using macro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: AUCMicro AUC score for multiclass classification using micro averaging. .. rubric:: Example >>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(AUCMicro().objective_function(y_true, y_pred), 0.9861111) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - AUC Micro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AUCMicro.calculate_percent_difference evalml.objectives.AUCMicro.is_defined_for_problem_type evalml.objectives.AUCMicro.objective_function evalml.objectives.AUCMicro.positive_only evalml.objectives.AUCMicro.score evalml.objectives.AUCMicro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for AUC score for multiclass classification using micro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: AUCWeighted AUC Score for multiclass classification using weighted averaging. .. rubric:: Example >>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.1, 0.0, 0.9], ... [0.1, 0.3, 0.6], ... [0.1, 0.2, 0.7], ... [0.6, 0.1, 0.3], ... [0.5, 0.2, 0.3]] >>> np.testing.assert_almost_equal(AUCWeighted().objective_function(y_true, y_pred), 0.4375) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - AUC Weighted * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.AUCWeighted.calculate_percent_difference evalml.objectives.AUCWeighted.is_defined_for_problem_type evalml.objectives.AUCWeighted.objective_function evalml.objectives.AUCWeighted.positive_only evalml.objectives.AUCWeighted.score evalml.objectives.AUCWeighted.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for AUC Score for multiclass classification using weighted averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: BalancedAccuracyBinary Balanced accuracy score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(BalancedAccuracyBinary().objective_function(y_true, y_pred), 0.60) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Balanced Accuracy Binary * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.BalancedAccuracyBinary.calculate_percent_difference evalml.objectives.BalancedAccuracyBinary.can_optimize_threshold evalml.objectives.BalancedAccuracyBinary.decision_function evalml.objectives.BalancedAccuracyBinary.is_defined_for_problem_type evalml.objectives.BalancedAccuracyBinary.objective_function evalml.objectives.BalancedAccuracyBinary.optimize_threshold evalml.objectives.BalancedAccuracyBinary.positive_only evalml.objectives.BalancedAccuracyBinary.score evalml.objectives.BalancedAccuracyBinary.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for accuracy score for balanced accuracy for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: BalancedAccuracyMulticlass Balanced accuracy score for multiclass classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(BalancedAccuracyMulticlass().objective_function(y_true, y_pred), 0.5555555) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Balanced Accuracy Multiclass * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.BalancedAccuracyMulticlass.calculate_percent_difference evalml.objectives.BalancedAccuracyMulticlass.is_defined_for_problem_type evalml.objectives.BalancedAccuracyMulticlass.objective_function evalml.objectives.BalancedAccuracyMulticlass.positive_only evalml.objectives.BalancedAccuracyMulticlass.score evalml.objectives.BalancedAccuracyMulticlass.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for accuracy score for balanced accuracy for multiclass classification. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: BinaryClassificationObjective Base class for all binary classification objectives. **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.BinaryClassificationObjective.calculate_percent_difference evalml.objectives.BinaryClassificationObjective.can_optimize_threshold evalml.objectives.BinaryClassificationObjective.decision_function evalml.objectives.BinaryClassificationObjective.expected_range evalml.objectives.BinaryClassificationObjective.greater_is_better evalml.objectives.BinaryClassificationObjective.is_bounded_like_percentage evalml.objectives.BinaryClassificationObjective.is_defined_for_problem_type evalml.objectives.BinaryClassificationObjective.name evalml.objectives.BinaryClassificationObjective.objective_function evalml.objectives.BinaryClassificationObjective.optimize_threshold evalml.objectives.BinaryClassificationObjective.perfect_score evalml.objectives.BinaryClassificationObjective.positive_only evalml.objectives.BinaryClassificationObjective.score evalml.objectives.BinaryClassificationObjective.score_needs_proba evalml.objectives.BinaryClassificationObjective.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: expected_range(cls) :property: Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1]. .. py:method:: greater_is_better(cls) :property: Returns a boolean determining if a greater score indicates better model performance. .. py:method:: is_bounded_like_percentage(cls) :property: Returns whether this objective is bounded between 0 and 1, inclusive. .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: name(cls) :property: Returns a name describing the objective. .. py:method:: objective_function(cls, y_true, y_predicted, y_train=None, X=None, sample_weight=None) :classmethod: :abstractmethod: Computes the relative value of the provided predictions compared to the actual labels, according a specified metric. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: Numerical value used to calculate score .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: perfect_score(cls) :property: Returns the score obtained by evaluating this objective on a perfect model. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: score_needs_proba(cls) :property: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation. .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: CostBenefitMatrix(true_positive, true_negative, false_positive, false_negative) Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow. :param true_positive: Cost associated with true positive predictions. :type true_positive: float :param true_negative: Cost associated with true negative predictions. :type true_negative: float :param false_positive: Cost associated with false positive predictions. :type false_positive: float :param false_negative: Cost associated with false negative predictions. :type false_negative: float **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - Cost Benefit Matrix * - **perfect_score** - None * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.CostBenefitMatrix.calculate_percent_difference evalml.objectives.CostBenefitMatrix.can_optimize_threshold evalml.objectives.CostBenefitMatrix.decision_function evalml.objectives.CostBenefitMatrix.is_defined_for_problem_type evalml.objectives.CostBenefitMatrix.objective_function evalml.objectives.CostBenefitMatrix.optimize_threshold evalml.objectives.CostBenefitMatrix.positive_only evalml.objectives.CostBenefitMatrix.score evalml.objectives.CostBenefitMatrix.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Calculates cost-benefit of the using the predicted and true values. :param y_predicted: Predicted labels. :type y_predicted: pd.Series :param y_true: True labels. :type y_true: pd.Series :param y_train: Ignored. :type y_train: pd.Series :param X: Ignored. :type X: pd.DataFrame :param sample_weight: Ignored. :type sample_weight: pd.DataFrame :returns: Cost-benefit matrix score :rtype: float .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: ExpVariance Explained variance score for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(ExpVariance().objective_function(y_true, y_pred), 0.7760736) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - ExpVariance * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.ExpVariance.calculate_percent_difference evalml.objectives.ExpVariance.is_defined_for_problem_type evalml.objectives.ExpVariance.objective_function evalml.objectives.ExpVariance.positive_only evalml.objectives.ExpVariance.score evalml.objectives.ExpVariance.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for explained variance score for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: F1 F1 score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(F1().objective_function(y_true, y_pred), 0.25) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - F1 * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.F1.calculate_percent_difference evalml.objectives.F1.can_optimize_threshold evalml.objectives.F1.decision_function evalml.objectives.F1.is_defined_for_problem_type evalml.objectives.F1.objective_function evalml.objectives.F1.optimize_threshold evalml.objectives.F1.positive_only evalml.objectives.F1.score evalml.objectives.F1.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for F1 score for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: F1Macro F1 score for multiclass classification using macro averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Macro().objective_function(y_true, y_pred), 0.5476190) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - F1 Macro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.F1Macro.calculate_percent_difference evalml.objectives.F1Macro.is_defined_for_problem_type evalml.objectives.F1Macro.objective_function evalml.objectives.F1Macro.positive_only evalml.objectives.F1Macro.score evalml.objectives.F1Macro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for F1 score for multiclass classification using macro averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: F1Micro F1 score for multiclass classification using micro averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Micro().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - F1 Micro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.F1Micro.calculate_percent_difference evalml.objectives.F1Micro.is_defined_for_problem_type evalml.objectives.F1Micro.objective_function evalml.objectives.F1Micro.positive_only evalml.objectives.F1Micro.score evalml.objectives.F1Micro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for F1 score for multiclass classification. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: F1Weighted F1 score for multiclass classification using weighted averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(F1Weighted().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - F1 Weighted * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.F1Weighted.calculate_percent_difference evalml.objectives.F1Weighted.is_defined_for_problem_type evalml.objectives.F1Weighted.objective_function evalml.objectives.F1Weighted.positive_only evalml.objectives.F1Weighted.score evalml.objectives.F1Weighted.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for F1 score for multiclass classification using weighted averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: FraudCost(retry_percentage=0.5, interchange_fee=0.02, fraud_payout_percentage=1.0, amount_col='amount') Score the percentage of money lost of the total transaction amount process due to fraud. :param retry_percentage: What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to 0.5. :type retry_percentage: float :param interchange_fee: How much of each successful transaction you pay. Between 0 and 1. Defaults to 0.02. :type interchange_fee: float :param fraud_payout_percentage: Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0. :type fraud_payout_percentage: float :param amount_col: Name of column in data that contains the amount. Defaults to "amount". :type amount_col: str **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - True * - **name** - Fraud Cost * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.FraudCost.calculate_percent_difference evalml.objectives.FraudCost.can_optimize_threshold evalml.objectives.FraudCost.decision_function evalml.objectives.FraudCost.is_defined_for_problem_type evalml.objectives.FraudCost.objective_function evalml.objectives.FraudCost.optimize_threshold evalml.objectives.FraudCost.positive_only evalml.objectives.FraudCost.score evalml.objectives.FraudCost.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, X, y_train=None, sample_weight=None) Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount. :param y_predicted: Predicted fraud labels. :type y_predicted: pd.Series :param y_true: True fraud labels. :type y_true: pd.Series :param y_train: Ignored. :type y_train: pd.Series :param X: Data with transaction amounts. :type X: pd.DataFrame :param sample_weight: Ignored. :type sample_weight: pd.DataFrame :returns: Amount lost to fraud per transaction. :rtype: float :raises ValueError: If amount_col is not a valid column in the input data. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:function:: get_all_objective_names() Get a list of the names of all objectives. :returns: Objective names :rtype: list (str) .. py:function:: get_core_objective_names() Get a list of all valid core objectives. :returns: Objective names. :rtype: list[str] .. py:function:: get_core_objectives(problem_type) Returns all core objective instances associated with the given problem type. Core objectives are designed to work out-of-the-box for any dataset. :param problem_type: Type of problem :type problem_type: str/ProblemTypes :returns: List of ObjectiveBase instances .. rubric:: Examples >>> for objective in get_core_objectives("regression"): ... print(objective.name) ExpVariance MaxError MedianAE MSE MAE R2 Root Mean Squared Error >>> for objective in get_core_objectives("binary"): ... print(objective.name) MCC Binary Log Loss Binary Gini AUC Precision F1 Balanced Accuracy Binary Accuracy Binary .. py:function:: get_default_recommendation_objectives(problem_type, imbalanced=False) Get the default recommendation score metrics for the given problem type. :param problem_type: Type of problem :type problem_type: str/ProblemType :param imbalanced: For multiclass problems, if the classes are imbalanced. Defaults to False :type imbalanced: boolean :returns: Set of string objective names that correspond to ObjectiveBase objectives .. py:function:: get_non_core_objectives() Get non-core objective classes. Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives before using them in AutoMLSearch. :returns: List of ObjectiveBase classes .. py:function:: get_objective(objective, return_instance=False, **kwargs) Returns the Objective class corresponding to a given objective name. :param objective: Name or instance of the objective class. :type objective: str or ObjectiveBase :param return_instance: Whether to return an instance of the objective. This only applies if objective is of type str. Note that the instance will be initialized with default arguments. :type return_instance: bool :param kwargs: Any keyword arguments to pass into the objective. Only used when return_instance=True. :type kwargs: Any :returns: ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid objective name, function will return the class corresponding to that name. If return_instance is True, an instance of that objective will be returned. :raises TypeError: If objective is None. :raises TypeError: If objective is not a string and not an instance of ObjectiveBase. :raises ObjectiveNotFoundError: If input objective is not a valid objective. :raises ObjectiveCreationError: If objective cannot be created properly. .. py:function:: get_optimization_objectives(problem_type) Get objectives for optimization. :param problem_type: Type of problem :type problem_type: str/ProblemTypes :returns: List of ObjectiveBase instances .. py:function:: get_ranking_objectives(problem_type) Get objectives for pipeline rankings. :param problem_type: Type of problem :type problem_type: str/ProblemTypes :returns: List of ObjectiveBase instances .. py:class:: Gini Gini coefficient for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Gini().objective_function(y_true, y_pred), 0.1428571) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - Gini * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.Gini.calculate_percent_difference evalml.objectives.Gini.can_optimize_threshold evalml.objectives.Gini.decision_function evalml.objectives.Gini.is_defined_for_problem_type evalml.objectives.Gini.objective_function evalml.objectives.Gini.optimize_threshold evalml.objectives.Gini.positive_only evalml.objectives.Gini.score evalml.objectives.Gini.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for Gini coefficient for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: LeadScoring(true_positives=1, false_positives=-1) Lead scoring. :param true_positives: Reward for a true positive. Defaults to 1. :type true_positives: int :param false_positives: Cost for a false positive. Should be negative. Defaults to -1. :type false_positives: int **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - Lead Scoring * - **perfect_score** - None * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.LeadScoring.calculate_percent_difference evalml.objectives.LeadScoring.can_optimize_threshold evalml.objectives.LeadScoring.decision_function evalml.objectives.LeadScoring.is_defined_for_problem_type evalml.objectives.LeadScoring.objective_function evalml.objectives.LeadScoring.optimize_threshold evalml.objectives.LeadScoring.positive_only evalml.objectives.LeadScoring.score evalml.objectives.LeadScoring.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Calculate the profit per lead. :param y_predicted: Predicted labels. :type y_predicted: pd.Series :param y_true: True labels. :type y_true: pd.Series :param y_train: Ignored. :type y_train: pd.Series :param X: Ignored. :type X: pd.DataFrame :param sample_weight: Ignored. :type sample_weight: pd.DataFrame :returns: Profit per lead :rtype: float .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: LogLossBinary Log Loss for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(LogLossBinary().objective_function(y_true, y_pred), 19.6601745) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Log Loss Binary * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.LogLossBinary.calculate_percent_difference evalml.objectives.LogLossBinary.can_optimize_threshold evalml.objectives.LogLossBinary.decision_function evalml.objectives.LogLossBinary.is_defined_for_problem_type evalml.objectives.LogLossBinary.objective_function evalml.objectives.LogLossBinary.optimize_threshold evalml.objectives.LogLossBinary.positive_only evalml.objectives.LogLossBinary.score evalml.objectives.LogLossBinary.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for log loss for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: LogLossMulticlass Log Loss for multiclass classification. .. rubric:: Example >>> y_true = [0, 1, 2, 0, 2, 1] >>> y_pred = [[0.7, 0.2, 0.1], ... [0.3, 0.5, 0.2], ... [0.1, 0.3, 0.6], ... [0.9, 0.1, 0.0], ... [0.3, 0.1, 0.6], ... [0.5, 0.5, 0.0]] >>> np.testing.assert_almost_equal(LogLossMulticlass().objective_function(y_true, y_pred), 0.4783301) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Log Loss Multiclass * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - True **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.LogLossMulticlass.calculate_percent_difference evalml.objectives.LogLossMulticlass.is_defined_for_problem_type evalml.objectives.LogLossMulticlass.objective_function evalml.objectives.LogLossMulticlass.positive_only evalml.objectives.LogLossMulticlass.score evalml.objectives.LogLossMulticlass.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for log loss for multiclass classification. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MAE Mean absolute error for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAE().objective_function(y_true, y_pred), 0.2727272) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - MAE * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MAE.calculate_percent_difference evalml.objectives.MAE.is_defined_for_problem_type evalml.objectives.MAE.objective_function evalml.objectives.MAE.positive_only evalml.objectives.MAE.score evalml.objectives.MAE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for mean absolute error for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MAPE Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage. Only valid for nonzero inputs. Otherwise, will throw a ValueError. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MAPE().objective_function(y_true, y_pred), 15.9848484) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Mean Absolute Percentage Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MAPE.calculate_percent_difference evalml.objectives.MAPE.is_defined_for_problem_type evalml.objectives.MAPE.objective_function evalml.objectives.MAPE.positive_only evalml.objectives.MAPE.score evalml.objectives.MAPE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for mean absolute percentage error for time series regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MASE Mean absolute scaled error for time series regression. Only valid if there exists a nonzero input in y_train. Otherwise, will throw a ValueError. .. rubric:: Example >>> y_train = pd.Series([5, 0.5, 4, 6, 3, 5, 2]) >>> y_true = pd.Series([3, -0.5, 2, 7, 2]) >>> y_pred = pd.Series([2.5, 0.0, 2, 8, 1.25]) >>> np.testing.assert_almost_equal(MASE().objective_function(y_true, y_pred, y_train), 0.18333333333333335) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Mean Absolute Scaled Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MASE.calculate_percent_difference evalml.objectives.MASE.is_defined_for_problem_type evalml.objectives.MASE.objective_function evalml.objectives.MASE.positive_only evalml.objectives.MASE.score evalml.objectives.MASE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train, X=None, sample_weight=None) Objective function for mean absolute scaled error for time series regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MaxError Maximum residual error for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MaxError().objective_function(y_true, y_pred), 1.0) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - MaxError * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MaxError.calculate_percent_difference evalml.objectives.MaxError.is_defined_for_problem_type evalml.objectives.MaxError.objective_function evalml.objectives.MaxError.positive_only evalml.objectives.MaxError.score evalml.objectives.MaxError.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for maximum residual error for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MCCBinary Matthews correlation coefficient for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(MCCBinary().objective_function(y_true, y_pred), 0.2390457) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - MCC Binary * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MCCBinary.calculate_percent_difference evalml.objectives.MCCBinary.can_optimize_threshold evalml.objectives.MCCBinary.decision_function evalml.objectives.MCCBinary.is_defined_for_problem_type evalml.objectives.MCCBinary.objective_function evalml.objectives.MCCBinary.optimize_threshold evalml.objectives.MCCBinary.positive_only evalml.objectives.MCCBinary.score evalml.objectives.MCCBinary.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for Matthews correlation coefficient for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: MCCMulticlass Matthews correlation coefficient for multiclass classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(MCCMulticlass().objective_function(y_true, y_pred), 0.325) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - MCC Multiclass * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MCCMulticlass.calculate_percent_difference evalml.objectives.MCCMulticlass.is_defined_for_problem_type evalml.objectives.MCCMulticlass.objective_function evalml.objectives.MCCMulticlass.positive_only evalml.objectives.MCCMulticlass.score evalml.objectives.MCCMulticlass.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for Matthews correlation coefficient for multiclass classification. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MeanSquaredLogError Mean squared log error for regression. Only valid for nonnegative inputs. Otherwise, will throw a ValueError. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MeanSquaredLogError().objective_function(y_true, y_pred), 0.0171353) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Mean Squared Log Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MeanSquaredLogError.calculate_percent_difference evalml.objectives.MeanSquaredLogError.is_defined_for_problem_type evalml.objectives.MeanSquaredLogError.objective_function evalml.objectives.MeanSquaredLogError.positive_only evalml.objectives.MeanSquaredLogError.score evalml.objectives.MeanSquaredLogError.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for mean squared log error for regression. .. py:method:: positive_only(self) If True, this objective is only valid for positive data. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MedianAE Median absolute error for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MedianAE().objective_function(y_true, y_pred), 0.25) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - MedianAE * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MedianAE.calculate_percent_difference evalml.objectives.MedianAE.is_defined_for_problem_type evalml.objectives.MedianAE.objective_function evalml.objectives.MedianAE.positive_only evalml.objectives.MedianAE.score evalml.objectives.MedianAE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for median absolute error for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MSE Mean squared error for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(MSE().objective_function(y_true, y_pred), 0.1590909) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - MSE * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MSE.calculate_percent_difference evalml.objectives.MSE.is_defined_for_problem_type evalml.objectives.MSE.objective_function evalml.objectives.MSE.positive_only evalml.objectives.MSE.score evalml.objectives.MSE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for mean squared error for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: MulticlassClassificationObjective Base class for all multiclass classification objectives. **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.MulticlassClassificationObjective.calculate_percent_difference evalml.objectives.MulticlassClassificationObjective.expected_range evalml.objectives.MulticlassClassificationObjective.greater_is_better evalml.objectives.MulticlassClassificationObjective.is_bounded_like_percentage evalml.objectives.MulticlassClassificationObjective.is_defined_for_problem_type evalml.objectives.MulticlassClassificationObjective.name evalml.objectives.MulticlassClassificationObjective.objective_function evalml.objectives.MulticlassClassificationObjective.perfect_score evalml.objectives.MulticlassClassificationObjective.positive_only evalml.objectives.MulticlassClassificationObjective.score evalml.objectives.MulticlassClassificationObjective.score_needs_proba evalml.objectives.MulticlassClassificationObjective.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: expected_range(cls) :property: Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1]. .. py:method:: greater_is_better(cls) :property: Returns a boolean determining if a greater score indicates better model performance. .. py:method:: is_bounded_like_percentage(cls) :property: Returns whether this objective is bounded between 0 and 1, inclusive. .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: name(cls) :property: Returns a name describing the objective. .. py:method:: objective_function(cls, y_true, y_predicted, y_train=None, X=None, sample_weight=None) :classmethod: :abstractmethod: Computes the relative value of the provided predictions compared to the actual labels, according a specified metric. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: Numerical value used to calculate score .. py:method:: perfect_score(cls) :property: Returns the score obtained by evaluating this objective on a perfect model. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: score_needs_proba(cls) :property: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation. .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:function:: normalize_objectives(objectives_to_normalize, max_objectives, min_objectives) Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective. :param objectives_to_normalize: A dictionary mapping objectives to values :type objectives_to_normalize: dict[str,float] :param max_objectives: The mapping of objectives to the maximum values for normalization :type max_objectives: dict[str,float] :param min_objectives: The mapping of objectives to the minimum values for normalization :type min_objectives: dict[str,float] :returns: A dictionary mapping objective names to their new normalized values .. py:class:: ObjectiveBase Base class for all objectives. **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **problem_types** - None **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.ObjectiveBase.calculate_percent_difference evalml.objectives.ObjectiveBase.expected_range evalml.objectives.ObjectiveBase.greater_is_better evalml.objectives.ObjectiveBase.is_bounded_like_percentage evalml.objectives.ObjectiveBase.is_defined_for_problem_type evalml.objectives.ObjectiveBase.name evalml.objectives.ObjectiveBase.objective_function evalml.objectives.ObjectiveBase.perfect_score evalml.objectives.ObjectiveBase.positive_only evalml.objectives.ObjectiveBase.score evalml.objectives.ObjectiveBase.score_needs_proba evalml.objectives.ObjectiveBase.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: expected_range(cls) :property: Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1]. .. py:method:: greater_is_better(cls) :property: Returns a boolean determining if a greater score indicates better model performance. .. py:method:: is_bounded_like_percentage(cls) :property: Returns whether this objective is bounded between 0 and 1, inclusive. .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: name(cls) :property: Returns a name describing the objective. .. py:method:: objective_function(cls, y_true, y_predicted, y_train=None, X=None, sample_weight=None) :classmethod: :abstractmethod: Computes the relative value of the provided predictions compared to the actual labels, according a specified metric. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: Numerical value used to calculate score .. py:method:: perfect_score(cls) :property: Returns the score obtained by evaluating this objective on a perfect model. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: score_needs_proba(cls) :property: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation. .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:function:: organize_objectives(problem_type, include=None, exclude=None, imbalanced=False) Generate objectives to consider, with optional modifications to the defaults. :param problem_type: Type of problem :type problem_type: str/ProblemType :param include: A list of objectives to include beyond the defaults. Defaults to None. :type include: list[str/ObjectiveBase] :param exclude: A list of objectives to exclude from the defaults. Defaults to None. :type exclude: list[str/ObjectiveBase] :param imbalanced: For multiclass problems, if the classes are imbalanced. Defaults to False :type imbalanced: boolean :returns: List of string objective names that correspond to ObjectiveBase objectives :raises ValueError: If any objectives to include or exclude are not valid for the problem type :raises ValueError: If an objective to exclude is not in the default objectives .. py:class:: Precision Precision score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Precision().objective_function(y_true, y_pred), 1.0) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Precision * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.Precision.calculate_percent_difference evalml.objectives.Precision.can_optimize_threshold evalml.objectives.Precision.decision_function evalml.objectives.Precision.is_defined_for_problem_type evalml.objectives.Precision.objective_function evalml.objectives.Precision.optimize_threshold evalml.objectives.Precision.positive_only evalml.objectives.Precision.score evalml.objectives.Precision.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for precision score for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: PrecisionMacro Precision score for multiclass classification using macro-averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMacro().objective_function(y_true, y_pred), 0.5555555) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Precision Macro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.PrecisionMacro.calculate_percent_difference evalml.objectives.PrecisionMacro.is_defined_for_problem_type evalml.objectives.PrecisionMacro.objective_function evalml.objectives.PrecisionMacro.positive_only evalml.objectives.PrecisionMacro.score evalml.objectives.PrecisionMacro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for precision score for multiclass classification using macro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: PrecisionMicro Precision score for multiclass classification using micro averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionMicro().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Precision Micro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.PrecisionMicro.calculate_percent_difference evalml.objectives.PrecisionMicro.is_defined_for_problem_type evalml.objectives.PrecisionMicro.objective_function evalml.objectives.PrecisionMicro.positive_only evalml.objectives.PrecisionMicro.score evalml.objectives.PrecisionMicro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for precision score for binary classification using micro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: PrecisionWeighted Precision score for multiclass classification using weighted averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(PrecisionWeighted().objective_function(y_true, y_pred), 0.5606060) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Precision Weighted * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.PrecisionWeighted.calculate_percent_difference evalml.objectives.PrecisionWeighted.is_defined_for_problem_type evalml.objectives.PrecisionWeighted.objective_function evalml.objectives.PrecisionWeighted.positive_only evalml.objectives.PrecisionWeighted.score evalml.objectives.PrecisionWeighted.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for precision score for multiclass classification using weighted averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: R2 Coefficient of determination for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(R2().objective_function(y_true, y_pred), 0.7638036) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - True * - **is_bounded_like_percentage** - False * - **name** - R2 * - **perfect_score** - 1 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.R2.calculate_percent_difference evalml.objectives.R2.is_defined_for_problem_type evalml.objectives.R2.objective_function evalml.objectives.R2.positive_only evalml.objectives.R2.score evalml.objectives.R2.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for coefficient of determination for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:function:: ranking_only_objectives() Get ranking-only objective classes. Ranking-only objectives are objectives that are useful for evaluating the performance of a model, but should not be used as an optimization objective during AutoMLSearch for various reasons. :returns: List of ObjectiveBase classes .. py:class:: Recall Recall score for binary classification. .. rubric:: Example >>> y_true = pd.Series([0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1]) >>> y_pred = pd.Series([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) >>> np.testing.assert_almost_equal(Recall().objective_function(y_true, y_pred), 0.1428571) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Recall * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.Recall.calculate_percent_difference evalml.objectives.Recall.can_optimize_threshold evalml.objectives.Recall.decision_function evalml.objectives.Recall.is_defined_for_problem_type evalml.objectives.Recall.objective_function evalml.objectives.Recall.optimize_threshold evalml.objectives.Recall.positive_only evalml.objectives.Recall.score evalml.objectives.Recall.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, threshold=0.5, X=None) Apply a learned threshold to predicted probabilities to get predicted classes. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series, np.ndarray :param threshold: Threshold used to make a prediction. Defaults to 0.5. :type threshold: float, optional :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: predictions .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for recall score for binary classification. .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: RecallMacro Recall score for multiclass classification using macro averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMacro().objective_function(y_true, y_pred), 0.5555555) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Recall Macro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RecallMacro.calculate_percent_difference evalml.objectives.RecallMacro.is_defined_for_problem_type evalml.objectives.RecallMacro.objective_function evalml.objectives.RecallMacro.positive_only evalml.objectives.RecallMacro.score evalml.objectives.RecallMacro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for recall score for multiclass classification using macro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: RecallMicro Recall score for multiclass classification using micro averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallMicro().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Recall Micro * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RecallMicro.calculate_percent_difference evalml.objectives.RecallMicro.is_defined_for_problem_type evalml.objectives.RecallMicro.objective_function evalml.objectives.RecallMicro.positive_only evalml.objectives.RecallMicro.score evalml.objectives.RecallMicro.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for recall score for multiclass classification using micro-averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: RecallWeighted Recall score for multiclass classification using weighted averaging. .. rubric:: Example >>> y_true = pd.Series([0, 1, 0, 2, 0, 1, 2, 1, 2, 0, 2]) >>> y_pred = pd.Series([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2]) >>> np.testing.assert_almost_equal(RecallWeighted().objective_function(y_true, y_pred), 0.5454545) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Recall Weighted * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.MULTICLASS, ProblemTypes.TIME_SERIES_MULTICLASS] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RecallWeighted.calculate_percent_difference evalml.objectives.RecallWeighted.is_defined_for_problem_type evalml.objectives.RecallWeighted.objective_function evalml.objectives.RecallWeighted.positive_only evalml.objectives.RecallWeighted.score evalml.objectives.RecallWeighted.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for recall score for multiclass classification using weighted averaging. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:function:: recommendation_score(objectives, prioritized_objective=None, custom_weights=None) Computes a recommendation score for a model given scores for a group of objectives. This recommendation score is a weighted average of the given objectives, by default all weighted equally. Passing in a prioritized objective will weight that objective with the prioritized weight, and all other objectives will split the remaining weight equally. :param objectives: A dictionary mapping objectives to their values. Objectives should be a float between 0 and 1, where higher is better. If the objective does not represent score this way, scores should first be normalized using the normalize_objectives function. :type objectives: dict[str,float] :param prioritized_objective: An optional name of a priority objective that should be given heavier weight (50% of the total) than the other objectives contributing to the score. Defaults to None, where all objectives are weighted equally. :type prioritized_objective: str :param custom_weights: A dictionary mapping objective names to corresponding weights between 0 and 1. If all objectives are listed, should add up to 1. If a subset of objectives are listed, should add up to less than 1, and remaining weight will be evenly distributed between the remaining objectives. Should not be used at the same time as prioritized_objective. :type custom_weights: dict[str,float] :returns: A value between 0 and 100 representing how strongly we recommend a pipeline given a set of evaluated objectives :raises ValueError: If the objective(s) to prioritize are not in the known objectives, or if the custom weight(s) are not a float between 0 and 1. .. py:class:: RegressionObjective Base class for all regression objectives. **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RegressionObjective.calculate_percent_difference evalml.objectives.RegressionObjective.expected_range evalml.objectives.RegressionObjective.greater_is_better evalml.objectives.RegressionObjective.is_bounded_like_percentage evalml.objectives.RegressionObjective.is_defined_for_problem_type evalml.objectives.RegressionObjective.name evalml.objectives.RegressionObjective.objective_function evalml.objectives.RegressionObjective.perfect_score evalml.objectives.RegressionObjective.positive_only evalml.objectives.RegressionObjective.score evalml.objectives.RegressionObjective.score_needs_proba evalml.objectives.RegressionObjective.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: expected_range(cls) :property: Returns the expected range of the objective, which is not necessarily the possible ranges. For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1]. .. py:method:: greater_is_better(cls) :property: Returns a boolean determining if a greater score indicates better model performance. .. py:method:: is_bounded_like_percentage(cls) :property: Returns whether this objective is bounded between 0 and 1, inclusive. .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: name(cls) :property: Returns a name describing the objective. .. py:method:: objective_function(cls, y_true, y_predicted, y_train=None, X=None, sample_weight=None) :classmethod: :abstractmethod: Computes the relative value of the provided predictions compared to the actual labels, according a specified metric. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: Numerical value used to calculate score .. py:method:: perfect_score(cls) :property: Returns the score obtained by evaluating this objective on a perfect model. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: score_needs_proba(cls) :property: Returns a boolean determining if the score() method needs probability estimates. This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation. .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: RootMeanSquaredError Root mean squared error for regression. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredError().objective_function(y_true, y_pred), 0.3988620) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Root Mean Squared Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RootMeanSquaredError.calculate_percent_difference evalml.objectives.RootMeanSquaredError.is_defined_for_problem_type evalml.objectives.RootMeanSquaredError.objective_function evalml.objectives.RootMeanSquaredError.positive_only evalml.objectives.RootMeanSquaredError.score evalml.objectives.RootMeanSquaredError.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for root mean squared error for regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: RootMeanSquaredLogError Root mean squared log error for regression. Only valid for nonnegative inputs. Otherwise, will throw a ValueError. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(RootMeanSquaredLogError().objective_function(y_true, y_pred), 0.13090204) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - None * - **greater_is_better** - False * - **is_bounded_like_percentage** - False * - **name** - Root Mean Squared Log Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.REGRESSION, ProblemTypes.TIME_SERIES_REGRESSION, ProblemTypes.MULTISERIES_TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.RootMeanSquaredLogError.calculate_percent_difference evalml.objectives.RootMeanSquaredLogError.is_defined_for_problem_type evalml.objectives.RootMeanSquaredLogError.objective_function evalml.objectives.RootMeanSquaredLogError.positive_only evalml.objectives.RootMeanSquaredLogError.score evalml.objectives.RootMeanSquaredLogError.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for root mean squared log error for regression. .. py:method:: positive_only(self) If True, this objective is only valid for positive data. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed. .. py:class:: SensitivityLowAlert(alert_rate=0.01) Sensitivity at Low Alert Rates. :param alert_rate: percentage of top scores to classify as high risk. :type alert_rate: float **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 1] * - **greater_is_better** - True * - **is_bounded_like_percentage** - True * - **name** - Sensitivity at Low Alert Rates * - **perfect_score** - 1.0 * - **problem_types** - [ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.SensitivityLowAlert.calculate_percent_difference evalml.objectives.SensitivityLowAlert.can_optimize_threshold evalml.objectives.SensitivityLowAlert.decision_function evalml.objectives.SensitivityLowAlert.is_defined_for_problem_type evalml.objectives.SensitivityLowAlert.objective_function evalml.objectives.SensitivityLowAlert.optimize_threshold evalml.objectives.SensitivityLowAlert.positive_only evalml.objectives.SensitivityLowAlert.score evalml.objectives.SensitivityLowAlert.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: can_optimize_threshold(cls) :property: Returns a boolean determining if we can optimize the binary classification objective threshold. This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true. :returns: Whether or not an objective can be optimized. :rtype: bool .. py:method:: decision_function(self, ypred_proba, **kwargs) Determine if an observation is high risk given an alert rate. :param ypred_proba: Predicted probabilities. :type ypred_proba: pd.Series :param \*\*kwargs: Additional abritrary parameters. :returns: Whether or not an observation is high risk given an alert rate. :rtype: pd.Series .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, **kwargs) Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class. :param y_true: True labels. :type y_true: pd.Series :param y_predicted: Predicted labels based on alert_rate. :type y_predicted: pd.Series :param \*\*kwargs: Additional abritrary parameters. :returns: sensitivity using the observations with the top scores as the predicted positive class. :rtype: float .. py:method:: optimize_threshold(self, ypred_proba, y_true, X=None) Learn a binary classification threshold which optimizes the current objective. :param ypred_proba: The classifier's predicted probabilities :type ypred_proba: pd.Series :param y_true: The ground truth for the predictions. :type y_true: pd.Series :param X: Any extra columns that are needed from training data. :type X: pd.DataFrame, optional :returns: Optimal threshold for this objective. :raises RuntimeError: If objective cannot be optimized. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validate inputs for scoring. .. py:class:: SMAPE Symmetric mean absolute percentage error for time series regression. Scaled by 100 to return a percentage. Only valid for nonzero inputs. Otherwise, will throw a ValueError. .. rubric:: Example >>> y_true = pd.Series([1.5, 2, 3, 1, 0.5, 1, 2.5, 2.5, 1, 0.5, 2]) >>> y_pred = pd.Series([1.5, 2.5, 2, 1, 0.5, 1, 3, 2.25, 0.75, 0.25, 1.75]) >>> np.testing.assert_almost_equal(SMAPE().objective_function(y_true, y_pred), 18.13652589) **Attributes** .. list-table:: :widths: 15 85 :header-rows: 0 * - **expected_range** - [0, 200] * - **greater_is_better** - False * - **is_bounded_like_percentage** - True * - **name** - Symmetric Mean Absolute Percentage Error * - **perfect_score** - 0.0 * - **problem_types** - [ProblemTypes.TIME_SERIES_REGRESSION] * - **score_needs_proba** - False **Methods** .. autoapisummary:: :nosignatures: evalml.objectives.SMAPE.calculate_percent_difference evalml.objectives.SMAPE.is_defined_for_problem_type evalml.objectives.SMAPE.objective_function evalml.objectives.SMAPE.positive_only evalml.objectives.SMAPE.score evalml.objectives.SMAPE.validate_inputs .. py:method:: calculate_percent_difference(cls, score, baseline_score) :classmethod: Calculate the percent difference between scores. :param score: A score. Output of the score method of this objective. :type score: float :param baseline_score: A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator. :type baseline_score: float :returns: The percent difference between the scores. Note that for objectives that can be interpreted as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score. :rtype: float .. py:method:: is_defined_for_problem_type(cls, problem_type) :classmethod: Returns whether or not an objective is defined for a problem type. .. py:method:: objective_function(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Objective function for symmetric mean absolute percentage error for time series regression. .. py:method:: positive_only(cls) If True, this objective is only valid for positive data. Defaults to False. .. py:method:: score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None) Returns a numerical score indicating performance based on the differences between the predicted and actual values. :param y_predicted: Predicted values of length [n_samples] :type y_predicted: pd.Series :param y_true: Actual class labels of length [n_samples] :type y_true: pd.Series :param y_train: Observed training values of length [n_samples] :type y_train: pd.Series :param X: Extra data of shape [n_samples, n_features] necessary to calculate score :type X: pd.DataFrame or np.ndarray :param sample_weight: Sample weights used in computing objective value result :type sample_weight: pd.DataFrame or np.ndarray :returns: score .. py:method:: validate_inputs(self, y_true, y_predicted) Validates the input based on a few simple checks. :param y_predicted: Predicted values of length [n_samples]. :type y_predicted: pd.Series, or pd.DataFrame :param y_true: Actual class labels of length [n_samples]. :type y_true: pd.Series :raises ValueError: If the inputs are malformed.