One of the key choices to make when training an ML model is what metric to choose by which to measure the efficacy of the model at learning the signal. Such metrics are useful for comparing how well the trained models generalize to new similar data.
This choice of metric is a key component of AutoML because it defines the cost function the AutoML search will seek to optimize. In EvalML, these metrics are called objectives. AutoML will seek to minimize (or maximize) the objective score as it explores more pipelines and parameters and will use the feedback from scoring pipelines to tune the available hyperparameters and continue the search. Therefore, it is critical to have an objective function that represents how the model will be applied in the intended domain of use.
EvalML supports a variety of objectives from traditional supervised ML including mean squared error for regression problems and cross entropy or area under the ROC curve for classification problems. EvalML also allows the user to define a custom objective using their domain expertise, so that AutoML can search for models which provide the most value for the user’s problem.
Use the get_objectives method to get a list of which objectives are included with EvalML for each problem type:
get_objectives
[1]:
from evalml.objectives import get_objectives from evalml.problem_types import ProblemTypes for objective in get_objectives(ProblemTypes.BINARY): print(objective.name)
Accuracy Binary Balanced Accuracy Binary F1 Precision AUC Log Loss Binary MCC Binary
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.11.2/lib/python3.7/site-packages/evalml/pipelines/components/transformers/preprocessing/text_featurization.py:35: RuntimeWarning: No text columns were given to TextFeaturizer, component will have no effect warnings.warn("No text columns were given to TextFeaturizer, component will have no effect", RuntimeWarning)
EvalML defines a base objective class for each problem type: RegressionObjective, BinaryClassificationObjective and MulticlassClassificationObjective. All EvalML objectives are a subclass of one of these.
RegressionObjective
BinaryClassificationObjective
MulticlassClassificationObjective
Often times, the objective function is very specific to the use-case or business problem. To get the right objective to optimize requires thinking through the decisions or actions that will be taken using the model and assigning a cost/benefit to doing that correctly or incorrectly based on known outcomes in the training data.
Once you have determined the objective for your business, you can provide that to EvalML to optimize by defining a custom objective function.
To create a custom objective class, we must define several elements:
name: The printable name of this objective.
name
objective_function: This function takes the predictions, true labels, and an optional reference to the inputs, and returns a score of how well the model performed.
objective_function
greater_is_better: True if a higher objective_function value represents a better solution, and otherwise False.
greater_is_better
True
False
score_needs_proba: Only for classification objectives. True if the objective is intended to function with predicted probabilities as opposed to predicted values (example: cross entropy for classifiers).
score_needs_proba
decision_function: Only for binary classification objectives. This function takes predicted probabilities that were output from the model and a binary classification threshold, and returns predicted values.
decision_function
To give a concrete example, let’s look at how the fraud detection objective function is built.
[2]:
from evalml.objectives.binary_classification_objective import BinaryClassificationObjective import pandas as pd class FraudCost(BinaryClassificationObjective): """Score the percentage of money lost of the total transaction amount process due to fraud""" name = "Fraud Cost" greater_is_better = False score_needs_proba = False def __init__(self, retry_percentage=.5, interchange_fee=.02, fraud_payout_percentage=1.0, amount_col='amount'): """Create instance of FraudCost Arguments: retry_percentage (float): What percentage of customers that will retry a transaction if it is declined. Between 0 and 1. Defaults to .5 interchange_fee (float): How much of each successful transaction you can collect. Between 0 and 1. Defaults to .02 fraud_payout_percentage (float): Percentage of fraud you will not be able to collect. Between 0 and 1. Defaults to 1.0 amount_col (str): Name of column in data that contains the amount. Defaults to "amount" """ self.retry_percentage = retry_percentage self.interchange_fee = interchange_fee self.fraud_payout_percentage = fraud_payout_percentage self.amount_col = amount_col def decision_function(self, ypred_proba, threshold=0.0, X=None): """Determine if a transaction is fraud given predicted probabilities, threshold, and dataframe with transaction amount Arguments: ypred_proba (pd.Series): Predicted probablities X (pd.DataFrame): Dataframe containing transaction amount threshold (float): Dollar threshold to determine if transaction is fraud Returns: pd.Series: Series of predicted fraud labels using X and threshold """ if not isinstance(X, pd.DataFrame): X = pd.DataFrame(X) if not isinstance(ypred_proba, pd.Series): ypred_proba = pd.Series(ypred_proba) transformed_probs = (ypred_proba.values * X[self.amount_col]) return transformed_probs > threshold def objective_function(self, y_true, y_predicted, X): """Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount Arguments: y_predicted (pd.Series): predicted fraud labels y_true (pd.Series): true fraud labels X (pd.DataFrame): dataframe with transaction amounts Returns: float: amount lost to fraud per transaction """ if not isinstance(X, pd.DataFrame): X = pd.DataFrame(X) if not isinstance(y_predicted, pd.Series): y_predicted = pd.Series(y_predicted) if not isinstance(y_true, pd.Series): y_true = pd.Series(y_true) # extract transaction using the amount columns in users data try: transaction_amount = X[self.amount_col] except KeyError: raise ValueError("`{}` is not a valid column in X.".format(self.amount_col)) # amount paid if transaction is fraud fraud_cost = transaction_amount * self.fraud_payout_percentage # money made from interchange fees on transaction interchange_cost = transaction_amount * (1 - self.retry_percentage) * self.interchange_fee # calculate cost of missing fraudulent transactions false_negatives = (y_true & ~y_predicted) * fraud_cost # calculate money lost from fees false_positives = (~y_true & y_predicted) * interchange_cost loss = false_negatives.sum() + false_positives.sum() loss_per_total_processed = loss / transaction_amount.sum() return loss_per_total_processed