{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Text Data with EvalML\n",
    "\n",
    "In this demo, we will show you how to use EvalML to build models which use text data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import evalml\n",
    "from evalml import AutoMLSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "We will be utilizing a dataset of SMS text messages, some of which are categorized as spam, and others which are not (\"ham\"). This dataset is originally from [Kaggle](https://www.kaggle.com/uciml/sms-spam-collection-dataset), but modified to produce a slightly more even distribution of spam to ham."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "import pandas as pd\n",
    "\n",
    "input_data = urlopen(\n",
    "    \"https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv\"\n",
    ")\n",
    "data = pd.read_csv(input_data)[:750]\n",
    "\n",
    "X = data.drop([\"Category\"], axis=1)\n",
    "y = data[\"Category\"]\n",
    "\n",
    "display(X.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ham vs spam distribution of the data is 3:1, so any machine learning model must get above 75% [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision#In_binary_classification) in order to perform better than a trivial baseline model which simply classifies everything as ham. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to properly utilize Woodwork's 'Natural Language' typing, we need to pass this argument in during initialization. Otherwise, this will be treated as an 'Unknown' type and dropped in the search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.ww.init(logical_types={\"Message\": \"NaturalLanguage\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search for best pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(\n",
    "    X, y, problem_type=\"binary\", test_size=0.2, random_seed=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EvalML uses [Woodwork](https://woodwork.alteryx.com/en/stable/) to automatically detect which columns are text columns, so you can run search normally, as you would if there was no text data. We can print out the logical type of the `Message` column and assert that it is indeed inferred as a natural language column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.ww"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the spam/ham labels are binary, we will use `AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')`. When we call `.search()`, the search for the best pipeline will begin. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl = AutoMLSearch(\n",
    "    X_train=X_train,\n",
    "    y_train=y_train,\n",
    "    problem_type=\"binary\",\n",
    "    max_batches=1,\n",
    "    optimize_thresholds=True,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "automl.search(interactive_plot=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View rankings and select pipeline\n",
    "\n",
    "Once the fitting process is done, we can see all of the pipelines that were searched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl.rankings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select the best pipeline we can call `automl.best_pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pipeline = automl.best_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe pipeline\n",
    "\n",
    "You can get more details about any pipeline, including how it performed on other objective functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl.describe_pipeline(automl.rankings.iloc[0][\"id\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pipeline.graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice above that there is a `Natural Language Featurizer` as the first step in the pipeline. AutoMLSearch uses the woodwork accessor to recognize that `'Message'` is a text column, and converts this text into numerical values that can be handled by the estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on holdout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can score the pipeline on the holdout data using the ranking objectives for binary classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = best_pipeline.score(\n",
    "    X_holdout, y_holdout, objectives=evalml.objectives.get_ranking_objectives(\"binary\")\n",
    ")\n",
    "print(f'Accuracy Binary: {scores[\"Accuracy Binary\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this model performs relatively well on this dataset, even on unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What does the Natural Language Featurizer do?\n",
    "\n",
    "Machine learning models cannot handle non-numeric data. Any text must be broken down into numeric features that provide useful information about that text. The Natural Natural Language Featurizer first normalizes your text by removing any punctuation and other non-alphanumeric characters and converting any capital letters to lowercase. From there, it passes the text into [featuretools](https://www.featuretools.com/)' [nlp_primitives](https://docs.featuretools.com/en/v0.16.0/api_reference.html#natural-language-processing-primitives) `dfs` search, resulting in several informative features that replace the original column in your dataset: Diversity Score, Mean Characters per Word, Polarity Score, LSA (Latent Semantic Analysis), Number of Characters, and Number of Words.\n",
    "\n",
    "**Diversity Score** is the ratio of unique words to total words.\n",
    "\n",
    "**Mean Characters per Word** is the average number of letters in each word.\n",
    "\n",
    "**Polarity Score** is a prediction of how \"polarized\" the text is, on a scale from -1 (extremely negative) to 1 (extremely positive).\n",
    "\n",
    "**Latent Semantic Analysis** is an abstract representation of how important each word is with respect to the entire text, reduced down into two values per text. While the other text features are each a single column, this feature adds two columns to your data, `LSA(column_name)[0]` and `LSA(column_name)[1]`.\n",
    "\n",
    "**Number of Characters** is the number of characters in the text.\n",
    "\n",
    "**Number of Words** is the number of words in the text.\n",
    "\n",
    "Let's see what this looks like with our spam/ham example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pipeline.input_feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the Natural Language Featurizer takes in a single \"Message\" column, but then the next component in the pipeline, the Imputer, receives five columns of input. These five columns are the result of featurizing the text-type \"Message\" column. Most importantly, these featurized columns are what ends up passed in to the estimator.\n",
    "\n",
    "If the dataset had any non-text columns, those would be left alone by this process. If the dataset had more than one text column, each would be broken into these five feature columns independently. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The features, more directly\n",
    "\n",
    "Rather than just checking the new column names, let's examine the output of this component directly. We can see this by running the component on its own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "natural_language_featurizer = evalml.pipelines.components.NaturalLanguageFeaturizer()\n",
    "X_featurized = natural_language_featurizer.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compare the input data to the output from the Natural Language Featurizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_featurized.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These numeric values now represent important information about the original text that the estimator at the end of the pipeline can successfully use to make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why encode text this way?\n",
    "\n",
    "To demonstrate the importance of text-specific modeling, let's train a model with the same dataset, without letting `AutoMLSearch` detect the text column. We can change this by explicitly setting the data type of the `'Message'` column in Woodwork to `Categorical` using the utility method `infer_feature_types`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from evalml.utils import infer_feature_types\n",
    "\n",
    "X = infer_feature_types(X, {\"Message\": \"Categorical\"})\n",
    "X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(\n",
    "    X, y, problem_type=\"binary\", test_size=0.2, random_seed=0\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_no_text = AutoMLSearch(\n",
    "    X_train=X_train,\n",
    "    y_train=y_train,\n",
    "    problem_type=\"binary\",\n",
    "    max_batches=1,\n",
    "    optimize_thresholds=True,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "automl_no_text.search(interactive_plot=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like before, we can look at the rankings and pick the best pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_no_text.rankings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pipeline_no_text = automl_no_text.best_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, changing the data type of the text column removed the `Natural Language Featurizer` from the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_pipeline_no_text.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_no_text.describe_pipeline(automl_no_text.rankings.iloc[0][\"id\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get standard performance metrics on holdout data\n",
    "scores = best_pipeline_no_text.score(\n",
    "    X_holdout, y_holdout, objectives=evalml.objectives.get_ranking_objectives(\"binary\")\n",
    ")\n",
    "print(f'Accuracy Binary: {scores[\"Accuracy Binary\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without the `Natural Language Featurizer`, the `'Message'` column was treated as a categorical column, and therefore the conversion of this text to numerical features happened in the `One Hot Encoder`. The best pipeline encoded the top 10 most frequent \"categories\" of these texts, meaning 10 text messages were one-hot encoded and all the others were dropped. Clearly, this removed almost all of the information from the dataset, as we can see the `best_pipeline_no_text` performs very similarly to randomly guessing \"ham\" in every case."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}