Building a Fraud Prediction Model with EvalML#
In this demo, we will build an optimized fraud prediction model using EvalML. To optimize the pipeline, we will set up an objective function to minimize the percentage of total transaction value lost to fraud. At the end of this demo, we also show you how introducing the right objective during the training results in a much better than using a generic machine learning metric like AUC.
[1]:
import evalml
from evalml import AutoMLSearch
from evalml.objectives import FraudCost
Configure “Cost of Fraud”#
To optimize the pipelines toward the specific business needs of this model, we can set our own assumptions for the cost of fraud. These parameters are
retry_percentage
- what percentage of customers will retry a transaction if it is declined?interchange_fee
- how much of each successful transaction do you collect?fraud_payout_percentage
- the percentage of fraud will you be unable to collectamount_col
- the column in the data the represents the transaction amount
Using these parameters, EvalML determines attempt to build a pipeline that will minimize the financial loss due to fraud.
[2]:
fraud_objective = FraudCost(
retry_percentage=0.5,
interchange_fee=0.02,
fraud_payout_percentage=0.75,
amount_col="amount",
)
Search for best pipeline#
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as the holdout set.
[3]:
X, y = evalml.demos.load_fraud(n_rows=5000)
Number of Features
Boolean 1
Categorical 6
Numeric 5
Number of training examples: 5000
Targets
False 86.20%
True 13.80%
Name: fraud, dtype: object
EvalML natively supports one-hot encoding. Here we keep 1 out of the 6 categorical columns to decrease computation time.
[4]:
cols_to_drop = ["datetime", "expiration_date", "country", "region", "provider"]
for col in cols_to_drop:
X.ww.pop(col)
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(
X, y, problem_type="binary", test_size=0.2, random_seed=0
)
X.ww
[4]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
card_id | int64 | Integer | ['numeric'] |
store_id | int64 | Integer | ['numeric'] |
amount | int64 | Integer | ['numeric'] |
currency | category | Categorical | ['category'] |
customer_present | bool | Boolean | [] |
lat | float64 | Double | ['numeric'] |
lng | float64 | Double | ['numeric'] |
Because the fraud labels are binary, we will use AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')
. When we call .search()
, the search for the best pipeline will begin.
[5]:
automl = AutoMLSearch(
X_train=X_train,
y_train=y_train,
problem_type="binary",
objective=fraud_objective,
additional_objectives=["auc", "f1", "precision"],
allowed_model_families=["random_forest", "linear_model"],
max_batches=1,
optimize_thresholds=True,
verbose=True,
)
automl.search(interactive_plot=False)
AutoMLSearch will use mean CV score to rank pipelines.
*****************************
* Beginning pipeline search *
*****************************
Optimizing for Fraud Cost.
Lower score is better.
Using SequentialEngine to train and score pipelines.
Searching up to 1 batches for a total of None pipelines.
Allowed model families:
Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
Starting cross validation
Finished cross validation - mean Fraud Cost: 0.790
*****************************
* Evaluating Batch Number 1 *
*****************************
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler + Standard Scaler:
Starting cross validation
Finished cross validation - mean Fraud Cost: 0.008
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
Starting cross validation
Finished cross validation - mean Fraud Cost: 0.009
Search finished after 00:08
Best pipeline: Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler + Standard Scaler
Best pipeline Fraud Cost: 0.008149