Preprocessing

Preprocessing utilities.

Submodules

Package Contents

Classes Summary

NoSplit

Does not split the training data into training and validation sets.

TimeSeriesSplit

Rolling Origin Cross Validation for time series problems.

TrainingValidationSplit

Split the training data into training and validation sets.

Functions

load_data

Load features and target from file.

number_of_features

Get the number of features of each specific dtype in a DataFrame.

split_data

Split data into train and test sets.

target_distribution

Get the target distributions.

Contents

evalml.preprocessing.load_data(path, index, target, n_rows=None, drop=None, verbose=True, **kwargs)[source]

Load features and target from file.

Parameters
  • path (str) – Path to file or a http/ftp/s3 URL.

  • index (str) – Column for index.

  • target (str) – Column for target.

  • n_rows (int) – Number of rows to return. Defaults to None.

  • drop (list) – List of columns to drop. Defaults to None.

  • verbose (bool) – If True, prints information about features and target. Defaults to True.

  • **kwargs – Other keyword arguments that should be passed to panda’s read_csv method.

Returns

Features matrix and target.

Return type

pd.DataFrame, pd.Series

class evalml.preprocessing.NoSplit(random_seed=0)[source]

Does not split the training data into training and validation sets.

All data is passed as the training set, test data is simply an array of None. To be used for future unsupervised learning, should not be used in any of the currently supported pipelines.

Parameters

random_seed (int) – The seed to use for random sampling. Defaults to 0. Not used.

Methods

get_n_splits

Return the number of splits of this object.

split

Divide the data into training and testing sets, where the testing set is empty.

static get_n_splits()[source]

Return the number of splits of this object.

Returns

Always returns 0.

Return type

int

split(self, X, y=None)[source]

Divide the data into training and testing sets, where the testing set is empty.

Parameters
  • X (pd.DataFrame) – Dataframe of points to split

  • y (pd.Series) – Series of points to split

Returns

Indices to split data into training and test set

Return type

list

evalml.preprocessing.number_of_features(dtypes)[source]

Get the number of features of each specific dtype in a DataFrame.

Parameters

dtypes (pd.Series) – DataFrame.dtypes to get the number of features for.

Returns

dtypes and the number of features for each input type.

Return type

pd.Series

Example

>>> X = pd.DataFrame()
>>> X["integers"] = [i for i in range(10)]
>>> X["floats"] = [float(i) for i in range(10)]
>>> X["strings"] = [str(i) for i in range(10)]
>>> X["booleans"] = [bool(i%2) for i in range(10)]

Lists the number of columns corresponding to each dtype.

>>> number_of_features(X.dtypes)
             Number of Features
Boolean                       1
Categorical                   1
Numeric                       2
evalml.preprocessing.split_data(X, y, problem_type, problem_configuration=None, test_size=0.2, random_seed=0)[source]

Split data into train and test sets.

Parameters
  • X (pd.DataFrame or np.ndarray) – data of shape [n_samples, n_features]

  • y (pd.Series, or np.ndarray) – target data of length [n_samples]

  • problem_type (str or ProblemTypes) – type of supervised learning problem. see evalml.problem_types.problemtype.all_problem_types for a full list.

  • problem_configuration (dict) – Additional parameters needed to configure the search. For example, in time series problems, values should be passed in for the time_index, gap, and max_delay variables.

  • test_size (float) – What percentage of data points should be included in the test set. Defaults to 0.2 (20%).

  • random_seed (int) – Seed for the random number generator. Defaults to 0.

Returns

Feature and target data each split into train and test sets.

Return type

pd.DataFrame, pd.DataFrame, pd.Series, pd.Series

Examples

>>> X = pd.DataFrame([1, 2, 3, 4, 5, 6], columns=["First"])
>>> y = pd.Series([8, 9, 10, 11, 12, 13])
...
>>> X_train, X_validation, y_train, y_validation = split_data(X, y, "regression", random_seed=42)
>>> X_train
   First
5      6
2      3
4      5
3      4
>>> X_validation
   First
0      1
1      2
>>> y_train
5    13
2    10
4    12
3    11
dtype: int64
>>> y_validation
0    8
1    9
dtype: int64
evalml.preprocessing.target_distribution(targets)[source]

Get the target distributions.

Parameters

targets (pd.Series) – Target data.

Returns

Target data and their frequency distribution as percentages.

Return type

pd.Series

Examples

>>> y = pd.Series([1, 2, 4, 1, 3, 3, 1, 2])
>>> target_distribution(y)
Targets
1    37.50%
2    25.00%
3    25.00%
4    12.50%
dtype: object
>>> y = pd.Series([True, False, False, False, True])
>>> target_distribution(y)
Targets
False    60.00%
True     40.00%
dtype: object
class evalml.preprocessing.TimeSeriesSplit(max_delay=0, gap=0, forecast_horizon=1, time_index=None, n_splits=3)[source]

Rolling Origin Cross Validation for time series problems.

The max_delay, gap, and forecast_horizon parameters are only used to validate that the requested split size is not too small given these parameters.

Parameters
  • max_delay (int) – Max delay value for feature engineering. Time series pipelines create delayed features from existing features. This process will introduce NaNs into the first max_delay number of rows. The splitter uses the last max_delay number of rows from the previous split as the first max_delay number of rows of the current split to avoid “throwing out” more data than in necessary. Defaults to 0.

  • gap (int) – Number of time units separating the data used to generate features and the data to forecast on. Defaults to 0.

  • forecast_horizon (int) – Number of time units to forecast. Defaults to 1.

  • time_index (str) – Name of the column containing the datetime information used to order the data. Defaults to None.

  • n_splits (int) – number of data splits to make. Defaults to 3.

Example

>>> import numpy as np
>>> import pandas as pd
...
>>> X = pd.DataFrame([i for i in range(10)], columns=["First"])
>>> y = pd.Series([i for i in range(10)])
...
>>> ts_split = TimeSeriesSplit(n_splits=4)
>>> generator_ = ts_split.split(X, y)
...
>>> first_split = next(generator_)
>>> assert (first_split[0] == np.array([0, 1])).all()
>>> assert (first_split[1] == np.array([2, 3])).all()
...
...
>>> second_split = next(generator_)
>>> assert (second_split[0] == np.array([0, 1, 2, 3])).all()
>>> assert (second_split[1] == np.array([4, 5])).all()
...
...
>>> third_split = next(generator_)
>>> assert (third_split[0] == np.array([0, 1, 2, 3, 4, 5])).all()
>>> assert (third_split[1] == np.array([6, 7])).all()
...
...
>>> fourth_split = next(generator_)
>>> assert (fourth_split[0] == np.array([0, 1, 2, 3, 4, 5, 6, 7])).all()
>>> assert (fourth_split[1] == np.array([8, 9])).all()

Methods

get_n_splits

Get the number of data splits.

split

Get the time series splits.

get_n_splits(self, X=None, y=None, groups=None)[source]

Get the number of data splits.

Parameters
  • X (pd.DataFrame, None) – Features to split.

  • y (pd.DataFrame, None) – Target variable to split. Defaults to None.

  • groups – Ignored but kept for compatibility with sklearn API. Defaults to None.

Returns

Number of splits.

split(self, X, y=None, groups=None)[source]

Get the time series splits.

X and y are assumed to be sorted in ascending time order. This method can handle passing in empty or None X and y data but note that X and y cannot be None or empty at the same time.

Parameters
  • X (pd.DataFrame, None) – Features to split.

  • y (pd.DataFrame, None) – Target variable to split. Defaults to None.

  • groups – Ignored but kept for compatibility with sklearn API. Defaults to None.

Yields

Iterator of (train, test) indices tuples.

Raises

ValueError – If one of the proposed splits would be empty.

class evalml.preprocessing.TrainingValidationSplit(test_size=None, train_size=None, shuffle=False, stratify=None, random_seed=0)[source]

Split the training data into training and validation sets.

Parameters
  • test_size (float) – What percentage of data points should be included in the validation set. Defalts to the complement of train_size if train_size is set, and 0.25 otherwise.

  • train_size (float) – What percentage of data points should be included in the training set. Defaults to the complement of test_size

  • shuffle (boolean) – Whether to shuffle the data before splitting. Defaults to False.

  • stratify (list) – Splits the data in a stratified fashion, using this argument as class labels. Defaults to None.

  • random_seed (int) – The seed to use for random sampling. Defaults to 0.

Examples

>>> import numpy as np
>>> import pandas as pd
...
>>> X = pd.DataFrame([i for i in range(10)], columns=["First"])
>>> y = pd.Series([i for i in range(10)])
...
>>> tv_split = TrainingValidationSplit()
>>> split_ = next(tv_split.split(X, y))
>>> assert (split_[0] == np.array([0, 1, 2, 3, 4, 5, 6])).all()
>>> assert (split_[1] == np.array([7, 8, 9])).all()
...
...
>>> tv_split = TrainingValidationSplit(test_size=0.5)
>>> split_ = next(tv_split.split(X, y))
>>> assert (split_[0] == np.array([0, 1, 2, 3, 4])).all()
>>> assert (split_[1] == np.array([5, 6, 7, 8, 9])).all()
...
...
>>> tv_split = TrainingValidationSplit(shuffle=True)
>>> split_ = next(tv_split.split(X, y))
>>> assert (split_[0] == np.array([9, 1, 6, 7, 3, 0, 5])).all()
>>> assert (split_[1] == np.array([2, 8, 4])).all()
...
...
>>> y = pd.Series([i % 3 for i in range(10)])
>>> tv_split = TrainingValidationSplit(shuffle=True, stratify=y)
>>> split_ = next(tv_split.split(X, y))
>>> assert (split_[0] == np.array([1, 9, 3, 2, 8, 6, 7])).all()
>>> assert (split_[1] == np.array([0, 4, 5])).all()

Methods

get_n_splits

Return the number of splits of this object.

split

Divide the data into training and testing sets.

static get_n_splits()[source]

Return the number of splits of this object.

Returns

Always returns 1.

Return type

int

split(self, X, y=None)[source]

Divide the data into training and testing sets.

Parameters
  • X (pd.DataFrame) – Dataframe of points to split

  • y (pd.Series) – Series of points to split

Returns

Indices to split data into training and test set

Return type

list