permutation_importance#

Permutation importance methods.

Module Contents#

Functions#

calculate_permutation_importance

Calculates permutation importance for features.

calculate_permutation_importance_one_column

Calculates permutation importance for one column in the original dataframe.

graph_permutation_importance

Generate a bar graph of the pipeline's permutation importance.

Contents#

evalml.model_understanding.permutation_importance.calculate_permutation_importance(pipeline, X, y, objective, n_repeats=5, n_jobs=None, random_seed=0)[source]#

Calculates permutation importance for features.

Parameters

  • pipeline (PipelineBase or subclass) – Fitted pipeline.

  • X (pd.DataFrame) – The input data used to score and compute permutation importance.

  • y (pd.Series) – The target data.

  • objective (str, ObjectiveBase) – Objective to score on.

  • n_repeats (int) – Number of times to permute a feature. Defaults to 5.

  • n_jobs (int or None) – Non-negative integer describing level of parallelism used for pipelines. None and 1 are equivalent. If set to -1, all CPUs are used. For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. Defaults to None.

  • random_seed (int) – Seed for the random number generator. Defaults to 0.

Returns

Mean feature importance scores over a number of shuffles.

Return type

pd.DataFrame

Raises

ValueError – If objective cannot be used with the given pipeline.

evalml.model_understanding.permutation_importance.calculate_permutation_importance_one_column(pipeline, X, y, col_name, objective, n_repeats=5, fast=True, precomputed_features=None, random_seed=0)[source]#

Calculates permutation importance for one column in the original dataframe.

Parameters

  • pipeline (PipelineBase or subclass) – Fitted pipeline.

  • X (pd.DataFrame) – The input data used to score and compute permutation importance.

  • y (pd.Series) – The target data.

  • col_name (str, int) – The column in X to calculate permutation importance for.

  • objective (str, ObjectiveBase) – Objective to score on.

  • n_repeats (int) – Number of times to permute a feature. Defaults to 5.

  • fast (bool) – Whether to use the fast method of calculating the permutation importance or not. Defaults to True.

  • precomputed_features (pd.DataFrame) – Precomputed features necessary to calculate permutation importance using the fast method. Defaults to None.

  • random_seed (int) – Seed for the random number generator. Defaults to 0.

Returns

Mean feature importance scores over a number of shuffles.

Return type

float

Raises

  • ValueError – If pipeline does not support fast permutation importance calculation.

  • ValueError – If precomputed_features is None.

evalml.model_understanding.permutation_importance.graph_permutation_importance(pipeline, X, y, objective, importance_threshold=0)[source]#

Generate a bar graph of the pipeline’s permutation importance.

Parameters

  • pipeline (PipelineBase or subclass) – Fitted pipeline.

  • X (pd.DataFrame) – The input data used to score and compute permutation importance.

  • y (pd.Series) – The target data.

  • objective (str, ObjectiveBase) – Objective to score on.

  • importance_threshold (float, optional) – If provided, graph features with a permutation importance whose absolute value is larger than importance_threshold. Defaults to 0.

Returns

plotly.Figure, a bar graph showing features and their respective permutation importance.

Raises

ValueError – If importance_threshold is not greater than or equal to 0.