API Reference#

Demo Datasets#

load_breast_cancer

Load breast cancer dataset. Binary classification problem.

load_churn

Load churn dataset, which can be used for binary classification problems.

load_diabetes

Load diabetes dataset. Used for regression problem.

load_fraud

Load credit card fraud dataset.

load_weather

Load the Australian daily-min-termperatures weather dataset.

load_wine

Load wine dataset. Multiclass problem.

Preprocessing#

Preprocessing Utils#

Utilities to preprocess data before using evalml.

load_data

Load features and target from file.

number_of_features

Get the number of features of each specific dtype in a DataFrame.

split_data

Split data into train and test sets.

target_distribution

Get the target distributions.

Data Splitters#

NoSplit

Does not split the training data into training and validation sets.

KFold

Wrapper class for sklearn's KFold splitter.

StratifiedKFold

Wrapper class for sklearn's Stratified KFold splitter.

TrainingValidationSplit

Split the training data into training and validation sets.

TimeSeriesSplit

Rolling Origin Cross Validation for time series problems.

Exceptions#

AutoMLSearchException

Exception raised when all pipelines in an automl batch return a score of NaN for the primary objective.

ComponentNotYetFittedError

An exception to be raised when predict/predict_proba/transform is called on a component without fitting first.

DataCheckInitError

Exception raised when a data check can't initialize with the parameters given.

MethodPropertyNotFoundError

Exception to raise when a class is does not have an expected method or property.

MissingComponentError

An exception raised when a component is not found in all_components().

NoPositiveLabelException

Exception when a particular classification label for the 'positive' class cannot be found in the column index or unique values.

ObjectiveCreationError

Exception when get_objective tries to instantiate an objective and required args are not provided.

ObjectiveNotFoundError

Exception to raise when specified objective does not exist.

PartialDependenceError

Exception raised for all errors that partial dependence can raise.

PipelineError

Exception raised for errors that can be raised when applying a pipeline.

PipelineNotFoundError

An exception raised when a particular pipeline is not found in automl search results.

PipelineNotYetFittedError

An exception to be raised when predict/predict_proba/transform is called on a pipeline without fitting first.

PipelineScoreError

An exception raised when a pipeline errors while scoring any objective in a list of objectives.

Warnings#

NullsInColumnWarning

Warning thrown when there are null values in the column of interest.

ParameterNotUsedWarning

Warning thrown when a pipeline parameter isn't used in a defined pipeline's component graph during initialization.

Error Codes#

PartialDependenceErrorCode

Enum identifying the type of error encountered in partial dependence.

PipelineErrorCodeEnum

Enum identifying the type of error encountered while applying a pipeline.

ValidationErrorCode

Enum identifying the type of error encountered in holdout validation.

AutoML#

AutoML Search Interface#

AutoMLSearch

Automated Pipeline search.

AutoML Utils#

get_default_primary_search_objective

Get the default primary search objective for a problem type.

get_threshold_tuning_info

Determine for a given automl config and pipeline what the threshold tuning objective should be and whether or not training data should be further split to achieve proper threshold tuning.

make_data_splitter

Given the training data and ML problem parameters, compute a data splitting method to use during AutoML search.

resplit_training_data

Further split the training data for a given pipeline. This is needed for binary pipelines in order to properly tune the threshold.

search

Given data and configuration, run an automl search.

search_iterative

Given data and configuration, run an automl search.

tune_binary_threshold

Tunes the threshold of a binary pipeline to the X and y thresholding data.

AutoML Algorithm Classes#

AutoMLAlgorithm

Base class for the AutoML algorithms which power EvalML.

DefaultAlgorithm

An automl algorithm that consists of two modes: fast and long, where fast is a subset of long.

IterativeAlgorithm

An automl algorithm which first fits a base round of pipelines with default parameters, then does a round of parameter tuning on each pipeline in order of performance.

AutoML Callbacks#

log_error_callback

Logs the exception thrown as an error.

raise_error_callback

Raises the exception thrown by the AutoMLSearch object.

silent_error_callback

No-op.

AutoML Engines#

CFEngine

The concurrent.futures (CF) engine.

DaskEngine

The dask engine.

EngineBase

Base class for EvalML engines.

SequentialEngine

The default engine for the AutoML search.

Pipelines#

Pipeline Base Classes#

BinaryClassificationPipeline

Pipeline subclass for all binary classification pipelines.

ClassificationPipeline

Pipeline subclass for all classification pipelines.

MulticlassClassificationPipeline

Pipeline subclass for all multiclass classification pipelines.

PipelineBase

Machine learning pipeline.

RegressionPipeline

Pipeline subclass for all regression pipelines.

TimeSeriesBinaryClassificationPipeline

Pipeline base class for time series binary classification problems.

TimeSeriesClassificationPipeline

Pipeline base class for time series classification problems.

TimeSeriesMulticlassClassificationPipeline

Pipeline base class for time series multiclass classification problems.

TimeSeriesRegressionPipeline

Pipeline base class for time series regression problems.

Pipeline Utils#

get_actions_from_option_defaults

Returns a list of actions based on the defaults parameters of each option in the input DataCheckActionOption list.

generate_pipeline_code

Creates and returns a string that contains the Python imports and code required for running the EvalML pipeline.

generate_pipeline_example

Creates and returns a string that contains the Python imports and code required for running the EvalML pipeline.

make_pipeline

Given input data, target data, an estimator class and the problem type, generates a pipeline class with a preprocessing chain which was recommended based on the inputs. The pipeline will be a subclass of the appropriate pipeline base class for the specified problem_type.

make_pipeline_from_actions

Creates a pipeline of components to address the input DataCheckAction list.

make_pipeline_from_data_check_output

Creates a pipeline of components to address warnings and errors output from running data checks. Uses all default suggestions.

rows_of_interest

Get the row indices of the data that are closest to the threshold. Works only for binary classification problems and pipelines.

Component Graphs#

ComponentGraph

Component graph for a pipeline as a directed acyclic graph (DAG).

Components#

Component Base Classes#

Components represent a step in a pipeline.

ComponentBase

Base class for all components.

Transformer

A component that may or may not need fitting that transforms data. These components are used before an estimator.

Estimator

A component that fits and predicts given data.

Component Utils#

allowed_model_families

List the model types allowed for a particular problem type.

estimator_unable_to_handle_nans

If True, provided estimator class is unable to handle NaN values as an input.

generate_component_code

Creates and returns a string that contains the Python imports and code required for running the EvalML component.

get_estimators

Returns the estimators allowed for a particular problem type.

handle_component_class

Standardizes input from a string name to a ComponentBase subclass if necessary.

make_balancing_dictionary

Makes dictionary for oversampler components. Find ratio of each class to the majority. If the ratio is smaller than the sampling_ratio, we want to oversample, otherwise, we don't want to sample at all, and we leave the data as is.

Transformers#

Transformers are components that take in data as input and output transformed data.

DateTimeFeaturizer

Transformer that can automatically extract features from datetime columns.

DFSTransformer

Featuretools DFS component that generates features for the input features.

DropColumns

Drops specified columns in input data.

DropNaNRowsTransformer

Transformer to drop rows with NaN values.

DropNullColumns

Transformer to drop features whose percentage of NaN values exceeds a specified threshold.

DropRowsTransformer

Transformer to drop rows specified by row indices.

EmailFeaturizer

Transformer that can automatically extract features from emails.

Imputer

Imputes missing data according to a specified imputation strategy.

LabelEncoder

A transformer that encodes target labels using values between 0 and num_classes - 1.

LinearDiscriminantAnalysis

Reduces the number of features by using Linear Discriminant Analysis.

LogTransformer

Applies a log transformation to the target data.

LSA

Transformer to calculate the Latent Semantic Analysis Values of text input.

NaturalLanguageFeaturizer

Transformer that can automatically featurize text columns using featuretools' nlp_primitives.

OneHotEncoder

A transformer that encodes categorical features in a one-hot numeric array.

OrdinalEncoder

A transformer that encodes ordinal features as an array of ordinal integers representing the relative order of categories.

Oversampler

SMOTE Oversampler component. Will automatically select whether to use SMOTE, SMOTEN, or SMOTENC based on inputs to the component.

PCA

Reduces the number of features by using Principal Component Analysis (PCA).

PerColumnImputer

Imputes missing data according to a specified imputation strategy per column.

PolynomialDecomposer

Removes trends and seasonality from time series by fitting a polynomial and moving average to the data.

ReplaceNullableTypes

Transformer to replace features with the new nullable dtypes with a dtype that is compatible in EvalML.

RFClassifierRFESelector

Selects relevant features using recursive feature elimination with a Random Forest Classifier.

RFClassifierSelectFromModel

Selects top features based on importance weights using a Random Forest classifier.

RFRegressorRFESelector

Selects relevant features using recursive feature elimination with a Random Forest Regressor.

RFRegressorSelectFromModel

Selects top features based on importance weights using a Random Forest regressor.

SelectByType

Selects columns by specified Woodwork logical type or semantic tag in input data.

SelectColumns

Selects specified columns in input data.

SimpleImputer

Imputes missing data according to a specified imputation strategy. Natural language columns are ignored.

StandardScaler

A transformer that standardizes input features by removing the mean and scaling to unit variance.

STLDecomposer

Removes trends and seasonality from time series using the STL algorithm.

TargetEncoder

A transformer that encodes categorical features into target encodings.

TargetImputer

Imputes missing target data according to a specified imputation strategy.

TimeSeriesFeaturizer

Transformer that delays input features and target variable for time series problems.

TimeSeriesImputer

Imputes missing data according to a specified timeseries-specific imputation strategy.

TimeSeriesRegularizer

Transformer that regularizes an inconsistently spaced datetime column.

Undersampler

Initializes an undersampling transformer to downsample the majority classes in the dataset.

URLFeaturizer

Transformer that can automatically extract features from URL.

Estimators#

Classifiers#

Classifiers are components that output a predicted class label.

BaselineClassifier

Classifier that predicts using the specified strategy.

CatBoostClassifier

CatBoost Classifier, a classifier that uses gradient-boosting on decision trees. CatBoost is an open-source library and natively supports categorical features.

DecisionTreeClassifier

Decision Tree Classifier.

ElasticNetClassifier

Elastic Net Classifier. Uses Logistic Regression with elasticnet penalty as the base estimator.

ExtraTreesClassifier

Extra Trees Classifier.

KNeighborsClassifier

K-Nearest Neighbors Classifier.

LightGBMClassifier

LightGBM Classifier.

LogisticRegressionClassifier

Logistic Regression Classifier.

RandomForestClassifier

Random Forest Classifier.

StackedEnsembleClassifier

Stacked Ensemble Classifier.

SVMClassifier

Support Vector Machine Classifier.

VowpalWabbitBinaryClassifier

Vowpal Wabbit Binary Classifier.

VowpalWabbitMulticlassClassifier

Vowpal Wabbit Multiclass Classifier.

XGBoostClassifier

XGBoost Classifier.

Regressors#

Regressors are components that output a predicted target value.

ARIMARegressor

Autoregressive Integrated Moving Average Model. The three parameters (p, d, q) are the AR order, the degree of differencing, and the MA order. More information here: https://www.statsmodels.org/devel/generated/statsmodels.tsa.arima.model.ARIMA.html.

BaselineRegressor

Baseline regressor that uses a simple strategy to make predictions. This is useful as a simple baseline regressor to compare with other regressors.

CatBoostRegressor

CatBoost Regressor, a regressor that uses gradient-boosting on decision trees. CatBoost is an open-source library and natively supports categorical features.

DecisionTreeRegressor

Decision Tree Regressor.

ElasticNetRegressor

Elastic Net Regressor.

ExponentialSmoothingRegressor

Holt-Winters Exponential Smoothing Forecaster.

ExtraTreesRegressor

Extra Trees Regressor.

LightGBMRegressor

LightGBM Regressor.

LinearRegressor

Linear Regressor.

ProphetRegressor

Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

RandomForestRegressor

Random Forest Regressor.

StackedEnsembleRegressor

Stacked Ensemble Regressor.

SVMRegressor

Support Vector Machine Regressor.

TimeSeriesBaselineEstimator

Time series estimator that predicts using the naive forecasting approach.

VowpalWabbitRegressor

Vowpal Wabbit Regressor.

XGBoostRegressor

XGBoost Regressor.

Model Understanding#

Metrics#

binary_objective_vs_threshold

Computes objective score as a function of potential binary classification decision thresholds for a fitted binary classification pipeline.

calculate_permutation_importance

Calculates permutation importance for features.

calculate_permutation_importance_one_column

Calculates permutation importance for one column in the original dataframe.

confusion_matrix

Confusion matrix for binary and multiclass classification.

find_confusion_matrix_per_thresholds

Gets the confusion matrix and histogram bins for each threshold as well as the best threshold per objective. Only works with Binary Classification Pipelines.

get_linear_coefficients

Returns a dataframe showing the features with the greatest predictive power for a linear model.

get_prediction_vs_actual_data

Combines y_true and y_pred into a single dataframe and adds a column for outliers. Used in graph_prediction_vs_actual().

get_prediction_vs_actual_over_time_data

Get the data needed for the prediction_vs_actual_over_time plot.

normalize_confusion_matrix

Normalizes a confusion matrix.

partial_dependence

Calculates one or two-way partial dependence.

precision_recall_curve

Given labels and binary classifier predicted probabilities, compute and return the data representing a precision-recall curve.

roc_curve

Given labels and classifier predicted probabilities, compute and return the data representing a Receiver Operating Characteristic (ROC) curve. Works with binary or multiclass problems.

t_sne

Get the transformed output after fitting X to the embedded space using t-SNE.

get_influential_features

Finds the most influential features as well as any detrimental features from a dataframe of feature importances.

readable_explanation

Outputs a human-readable explanation of trained pipeline behavior.

Visualization Methods#

graph_binary_objective_vs_threshold

Generates a plot graphing objective score vs. decision thresholds for a fitted binary classification pipeline.

graph_confusion_matrix

Generate and display a confusion matrix plot.

graph_partial_dependence

Create an one-way or two-way partial dependence plot.

graph_permutation_importance

Generate a bar graph of the pipeline's permutation importance.

graph_precision_recall_curve

Generate and display a precision-recall plot.

graph_prediction_vs_actual

Generate a scatter plot comparing the true and predicted values. Used for regression plotting.

graph_prediction_vs_actual_over_time

Plot the target values and predictions against time on the x-axis.

graph_roc_curve

Generate and display a Receiver Operating Characteristic (ROC) plot for binary and multiclass classification problems.

graph_t_sne

Plot high dimensional data into lower dimensional space using t-SNE.

Prediction Explanations#

explain_predictions

Creates a report summarizing the top contributing features for each data point in the input features.

explain_predictions_best_worst

Creates a report summarizing the top contributing features for the best and worst points in the dataset as measured by error to true labels.

Objectives#

Objective Base Classes#

ObjectiveBase

Base class for all objectives.

BinaryClassificationObjective

Base class for all binary classification objectives.

MulticlassClassificationObjective

Base class for all multiclass classification objectives.

RegressionObjective

Base class for all regression objectives.

Domain-Specific Objectives#

CostBenefitMatrix

Score using a cost-benefit matrix. Scores quantify the benefits of a given value, so greater numeric scores represents a better score. Costs and scores can be negative, indicating that a value is not beneficial. For example, in the case of monetary profit, a negative cost and/or score represents loss of cash flow.

FraudCost

Score the percentage of money lost of the total transaction amount process due to fraud.

LeadScoring

Lead scoring.

SensitivityLowAlert

Sensitivity at Low Alert Rates.

Classification Objectives#

AccuracyBinary

Accuracy score for binary classification.

AccuracyMulticlass

Accuracy score for multiclass classification.

AUC

AUC score for binary classification.

AUCMacro

AUC score for multiclass classification using macro averaging.

AUCMicro

AUC score for multiclass classification using micro averaging.

AUCWeighted

AUC Score for multiclass classification using weighted averaging.

Gini

Gini coefficient for binary classification.

BalancedAccuracyBinary

Balanced accuracy score for binary classification.

BalancedAccuracyMulticlass

Balanced accuracy score for multiclass classification.

F1

F1 score for binary classification.

F1Micro

F1 score for multiclass classification using micro averaging.

F1Macro

F1 score for multiclass classification using macro averaging.

F1Weighted

F1 score for multiclass classification using weighted averaging.

LogLossBinary

Log Loss for binary classification.

LogLossMulticlass

Log Loss for multiclass classification.

MCCBinary

Matthews correlation coefficient for binary classification.

MCCMulticlass

Matthews correlation coefficient for multiclass classification.

Precision

Precision score for binary classification.

PrecisionMicro

Precision score for multiclass classification using micro averaging.

PrecisionMacro

Precision score for multiclass classification using macro-averaging.

PrecisionWeighted

Precision score for multiclass classification using weighted averaging.

Recall

Recall score for binary classification.

RecallMicro

Recall score for multiclass classification using micro averaging.

RecallMacro

Recall score for multiclass classification using macro averaging.

RecallWeighted

Recall score for multiclass classification using weighted averaging.

Regression Objectives#

ExpVariance

Explained variance score for regression.

MAE

Mean absolute error for regression.

MASE

Mean absolute scaled error for time series regression.

MAPE

Mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

SMAPE

Symmetric mean absolute percentage error for time series regression. Scaled by 100 to return a percentage.

MSE

Mean squared error for regression.

MeanSquaredLogError

Mean squared log error for regression.

MedianAE

Median absolute error for regression.

MaxError

Maximum residual error for regression.

R2

Coefficient of determination for regression.

RootMeanSquaredError

Root mean squared error for regression.

RootMeanSquaredLogError

Root mean squared log error for regression.

Objective Utils#

get_all_objective_names

Get a list of the names of all objectives.

get_core_objectives

Returns all core objective instances associated with the given problem type.

get_core_objective_names

Get a list of all valid core objectives.

get_default_recommendation_objectives

Get the default recommendation score metrics for the given problem type.

get_non_core_objectives

Get non-core objective classes.

get_objective

Returns the Objective class corresponding to a given objective name.

get_optimization_objectives

Get objectives for optimization.

get_ranking_objectives

Get objectives for pipeline rankings.

normalize_objectives

Converts objectives from a [0, inf) scale to [0, 1] given a max and min for each objective.

organize_objectives

Generate objectives to consider, with optional modifications to the defaults.

ranking_only_objectives

Get ranking-only objective classes.

recommendation_score

Computes a recommendation score for a model given scores for a group of objectives.

Problem Types#

detect_problem_type

Determine the type of problem is being solved based on the targets (binary vs multiclass classification, regression). Ignores missing and null data.

handle_problem_types

Handles problem_type by either returning the ProblemTypes or converting from a str.

is_binary

Determines if the provided problem_type is a binary classification problem type.

is_classification

Determines if the provided problem_type is a classification problem type.

is_multiclass

Determines if the provided problem_type is a multiclass classification problem type.

is_regression

Determines if the provided problem_type is a regression problem type.

is_time_series

Determines if the provided problem_type is a time series problem type.

ProblemTypes

Enum defining the supported types of machine learning problems.

Model Family#

handle_model_family

Handles model_family by either returning the ModelFamily or converting from a string.

ModelFamily

Enum for family of machine learning models.

Tuners#

Tuner

Base Tuner class.

SKOptTuner

Bayesian Optimizer.

GridSearchTuner

Grid Search Optimizer, which generates all of the possible points to search for using a grid.

RandomSearchTuner

Random Search Optimizer.

Data Checks#

Data Check Classes#

ClassImbalanceDataCheck

Check if any of the target labels are imbalanced, or if the number of values for each target are below 2 times the number of CV folds. Use for classification problems.

DateTimeFormatDataCheck

Check if the datetime column has equally spaced intervals and is monotonically increasing or decreasing in order to be supported by time series estimators.

IDColumnsDataCheck

Check if any of the features are likely to be ID columns.

InvalidTargetDataCheck

Check if the target data is considered invalid.

MulticollinearityDataCheck

Check if any set features are likely to be multicollinear.

NoVarianceDataCheck

Check if the target or any of the features have no variance.

NullDataCheck

Check if there are any highly-null numerical, boolean, categorical, natural language, and unknown columns and rows in the input.

OutliersDataCheck

Checks if there are any outliers in input data by using IQR to determine score anomalies.

SparsityDataCheck

Check if there are any columns with sparsely populated values in the input.

TargetDistributionDataCheck

Check if the target data contains certain distributions that may need to be transformed prior training to improve model performance. Uses the Shapiro-Wilks test when the dataset is <=5000 samples, otherwise uses Jarque-Bera.

TargetLeakageDataCheck

Check if any of the features are highly correlated with the target by using mutual information, Pearson correlation, and other correlation metrics.

TimeSeriesParametersDataCheck

Checks whether the time series parameters are compatible with data splitting.

TimeSeriesSplittingDataCheck

Checks whether the time series target data is compatible with splitting.

UniquenessDataCheck

Check if there are any columns in the input that are either too unique for classification problems or not unique enough for regression problems.

DataCheck

Base class for all data checks.

DataChecks

A collection of data checks.

DefaultDataChecks

A collection of basic data checks that is used by AutoML by default.

Data Check Messages#

DataCheckMessage

Base class for a message returned by a DataCheck, tagged by name.

DataCheckError

DataCheckMessage subclass for errors returned by data checks.

DataCheckWarning

DataCheckMessage subclass for warnings returned by data checks.

Data Check Message Types#

DataCheckMessageType

Enum for type of data check message: WARNING or ERROR.

Data Check Message Codes#

DataCheckMessageCode

Enum for data check message code.

Data Check Actions#

DataCheckAction

A recommended action returned by a DataCheck.

DataCheckActionCode

Enum for data check action code.

DataCheckActionOption

A recommended action option returned by a DataCheck.

Utils#

General Utils#

convert_to_seconds

Converts a string describing a length of time to its length in seconds.

downcast_nullable_types

Downcasts IntegerNullable, BooleanNullable types to Double, Boolean in order to support certain estimators like ARIMA, CatBoost, and LightGBM.

drop_rows_with_nans

Drop rows that have any NaNs in all dataframes or series.

get_importable_subclasses

Get importable subclasses of a base class. Used to list all of our estimators, transformers, components and pipelines dynamically.

get_logger

Get the logger with the associated name.

get_time_index

Determines the column in the given data that should be used as the time index.

import_or_raise

Attempts to import the requested library by name. If the import fails, raises an ImportError or warning.

infer_feature_types

Create a Woodwork structure from the given list, pandas, or numpy input, with specified types for columns. If a column's type is not specified, it will be inferred by Woodwork.

is_all_numeric

Checks if the given DataFrame contains only numeric values.

get_random_state

Generates a numpy.random.RandomState instance using seed.

get_random_seed

Given a numpy.random.RandomState object, generate an int representing a seed value for another random number generator. Or, if given an int, return that int.

pad_with_nans

Pad the beginning num_to_pad rows with nans.

safe_repr

Convert the given value into a string that can safely be used for repr.

save_plot

Saves fig to filepath if specified, or to a default location if not.