Simple Sentence Similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Simple Sentence Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word embeddings have become widespread in Natural Language Processing. They allow us to easily compute the semantic similarity between two words, or to find the words most similar to a target word. However, in many applications we're more interested in the similarity between two sentences or short texts. In this notebook, I compare some simple ways of computing sentence similarity and investigate how they perform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STS Benchmark\n",
    "\n",
    "The STS Benchmark brings together the English data from the SemEval sentence similarity tasks between 2012 and 2017. The data is split in training, development and test data: http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy\n",
    "import math\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "def load_sts_dataset(filename):\n",
    "    # Loads a subset of the STS dataset into a DataFrame. In particular both\n",
    "    # sentences and their human rated similarity score.\n",
    "    sent_pairs = []\n",
    "    with tf.gfile.GFile(filename, \"r\") as f:\n",
    "        for line in f:\n",
    "            ts = line.strip().split(\"\\t\")\n",
    "            sent_pairs.append((ts[5], ts[6], float(ts[4])))\n",
    "    return pd.DataFrame(sent_pairs, columns=[\"sent_1\", \"sent_2\", \"sim\"])\n",
    "\n",
    "\n",
    "def download_and_load_sts_data():\n",
    "    sts_dataset = tf.keras.utils.get_file(\n",
    "        fname=\"Stsbenchmark.tar.gz\",\n",
    "        origin=\"http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz\",\n",
    "        extract=True)\n",
    "\n",
    "    sts_dev = load_sts_dataset(os.path.join(os.path.dirname(sts_dataset), \"stsbenchmark\", \"sts-dev.csv\"))\n",
    "    sts_test = load_sts_dataset(os.path.join(os.path.dirname(sts_dataset), \"stsbenchmark\", \"sts-test.csv\"))\n",
    "\n",
    "    return sts_dev, sts_test\n",
    "\n",
    "sts_dev, sts_test = download_and_load_sts_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sent_1</th>\n",
       "      <th>sent_2</th>\n",
       "      <th>sim</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A girl is styling her hair.</td>\n",
       "      <td>A girl is brushing her hair.</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A group of men play soccer on the beach.</td>\n",
       "      <td>A group of boys are playing soccer on the beach.</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>One woman is measuring another woman's ankle.</td>\n",
       "      <td>A woman measures another woman's ankle.</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A man is cutting up a cucumber.</td>\n",
       "      <td>A man is slicing a cucumber.</td>\n",
       "      <td>4.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A man is playing a harp.</td>\n",
       "      <td>A man is playing a keyboard.</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          sent_1  \\\n",
       "0                    A girl is styling her hair.   \n",
       "1       A group of men play soccer on the beach.   \n",
       "2  One woman is measuring another woman's ankle.   \n",
       "3                A man is cutting up a cucumber.   \n",
       "4                       A man is playing a harp.   \n",
       "\n",
       "                                             sent_2  sim  \n",
       "0                      A girl is brushing her hair.  2.5  \n",
       "1  A group of boys are playing soccer on the beach.  3.6  \n",
       "2           A woman measures another woman's ankle.  5.0  \n",
       "3                      A man is slicing a cucumber.  4.2  \n",
       "4                      A man is playing a keyboard.  1.5  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sts_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SICK data\n",
    "\n",
    "The SICK dataset contains 10,000 English sentence pairs labelled with their semantic relatedness and entailment relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "def download_sick(f): \n",
    "\n",
    "    response = requests.get(f).text\n",
    "\n",
    "    lines = response.split(\"\\n\")[1:]\n",
    "    lines = [l.split(\"\\t\") for l in lines if len(l) > 0]\n",
    "    lines = [l for l in lines if len(l) == 5]\n",
    "\n",
    "    df = pd.DataFrame(lines, columns=[\"idx\", \"sent_1\", \"sent_2\", \"sim\", \"label\"])\n",
    "    df['sim'] = pd.to_numeric(df['sim'])\n",
    "    return df\n",
    "    \n",
    "sick_train = download_sick(\"https://raw.githubusercontent.com/alvations/stasis/master/SICK-data/SICK_train.txt\")\n",
    "sick_dev = download_sick(\"https://raw.githubusercontent.com/alvations/stasis/master/SICK-data/SICK_trial.txt\")\n",
    "sick_test = download_sick(\"https://raw.githubusercontent.com/alvations/stasis/master/SICK-data/SICK_test_annotated.txt\")\n",
    "sick_all = sick_train.append(sick_test).append(sick_dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>idx</th>\n",
       "      <th>sent_1</th>\n",
       "      <th>sent_2</th>\n",
       "      <th>sim</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>A group of kids is playing in a yard and an ol...</td>\n",
       "      <td>A group of boys in a yard is playing and a man...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>A group of children is playing in the house an...</td>\n",
       "      <td>A group of kids is playing in a yard and an ol...</td>\n",
       "      <td>3.2</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>The young boys are playing outdoors and the ma...</td>\n",
       "      <td>The kids are playing outdoors near a man with ...</td>\n",
       "      <td>4.7</td>\n",
       "      <td>ENTAILMENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>The kids are playing outdoors near a man with ...</td>\n",
       "      <td>A group of kids is playing in a yard and an ol...</td>\n",
       "      <td>3.4</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9</td>\n",
       "      <td>The young boys are playing outdoors and the ma...</td>\n",
       "      <td>A group of kids is playing in a yard and an ol...</td>\n",
       "      <td>3.7</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  idx                                             sent_1  \\\n",
       "0   1  A group of kids is playing in a yard and an ol...   \n",
       "1   2  A group of children is playing in the house an...   \n",
       "2   3  The young boys are playing outdoors and the ma...   \n",
       "3   5  The kids are playing outdoors near a man with ...   \n",
       "4   9  The young boys are playing outdoors and the ma...   \n",
       "\n",
       "                                              sent_2  sim       label  \n",
       "0  A group of boys in a yard is playing and a man...  4.5     NEUTRAL  \n",
       "1  A group of kids is playing in a yard and an ol...  3.2     NEUTRAL  \n",
       "2  The kids are playing outdoors near a man with ...  4.7  ENTAILMENT  \n",
       "3  A group of kids is playing in a yard and an ol...  3.4     NEUTRAL  \n",
       "4  A group of kids is playing in a yard and an ol...  3.7     NEUTRAL  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sick_all[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "First we need to do some preparation: some of our models require the sentences to be tokenized, some do not. For that reason we'll make a simple Sentence class where we keep both the raw sentence and the tokenized sentence. The individual methods below will then pick the input they need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "STOP = set(nltk.corpus.stopwords.words(\"english\"))\n",
    "\n",
    "class Sentence:\n",
    "    \n",
    "    def __init__(self, sentence):\n",
    "        self.raw = sentence\n",
    "        normalized_sentence = sentence.replace(\"‘\", \"'\").replace(\"’\", \"'\")\n",
    "        self.tokens = [t.lower() for t in nltk.word_tokenize(normalized_sentence)]\n",
    "        self.tokens_without_stop = [t for t in self.tokens if t not in STOP]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to use the popular [Gensim](https://radimrehurek.com/gensim/) library to load two sets of widely used pre-trained word embeddings: \n",
    "[word2vec](https://www.tensorflow.org/tutorials/word2vec) and [GloVe](https://nlp.stanford.edu/projects/glove/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "\n",
    "PATH_TO_WORD2VEC = os.path.expanduser(\"~/data/word2vec/GoogleNews-vectors-negative300.bin\")\n",
    "PATH_TO_GLOVE = os.path.expanduser(\"~/data/glove/glove.840B.300d.txt\")\n",
    "\n",
    "word2vec = gensim.models.KeyedVectors.load_word2vec_format(PATH_TO_WORD2VEC, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load Glove, we have to convert the downloaded GloVe file to word2vec format and then load the embeddings into a Gensim model. This will take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_file = \"/tmp/glove.840B.300d.w2v.txt\"\n",
    "glove2word2vec(PATH_TO_GLOVE, tmp_file)\n",
    "glove = gensim.models.KeyedVectors.load_word2vec_format(tmp_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, in order to compute weighted averages of word embeddings later, we are going to load a file with word frequencies. These word frequencies have been collected from Wikipedia and saved in a tab-separated file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "PATH_TO_FREQUENCIES_FILE = \"data/sentence_similarity/frequencies.tsv\"\n",
    "PATH_TO_DOC_FREQUENCIES_FILE = \"data/sentence_similarity/doc_frequencies.tsv\"\n",
    "\n",
    "def read_tsv(f):\n",
    "    frequencies = {}\n",
    "    with open(f) as tsv:\n",
    "        tsv_reader = csv.reader(tsv, delimiter=\"\\t\")\n",
    "        for row in tsv_reader: \n",
    "            frequencies[row[0]] = int(row[1])\n",
    "        \n",
    "    return frequencies\n",
    "        \n",
    "frequencies = read_tsv(PATH_TO_FREQUENCIES_FILE)\n",
    "doc_frequencies = read_tsv(PATH_TO_DOC_FREQUENCIES_FILE)\n",
    "doc_frequencies[\"NUM_DOCS\"] = 1288431\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline\n",
    "\n",
    "As our baseline, we're going for the simplest way of computing sentence embeddings: just take the embeddings of the words in the sentence (minus the stopwords), and compute their average, weighted by the sentence frequency of each word. \n",
    "\n",
    "We then use the cosine to calculate the similarity between two sentence embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from collections import Counter\n",
    "import math\n",
    "\n",
    "def run_avg_benchmark(sentences1, sentences2, model=None, use_stoplist=False, doc_freqs=None): \n",
    "\n",
    "    if doc_freqs is not None:\n",
    "        N = doc_freqs[\"NUM_DOCS\"]\n",
    "    \n",
    "    sims = []\n",
    "    for (sent1, sent2) in zip(sentences1, sentences2):\n",
    "    \n",
    "        tokens1 = sent1.tokens_without_stop if use_stoplist else sent1.tokens\n",
    "        tokens2 = sent2.tokens_without_stop if use_stoplist else sent2.tokens\n",
    "\n",
    "        tokens1 = [token for token in tokens1 if token in model]\n",
    "        tokens2 = [token for token in tokens2 if token in model]\n",
    "        \n",
    "        if len(tokens1) == 0 or len(tokens2) == 0:\n",
    "            sims.append(0)\n",
    "            continue\n",
    "        \n",
    "        tokfreqs1 = Counter(tokens1)\n",
    "        tokfreqs2 = Counter(tokens2)\n",
    "        \n",
    "        weights1 = [tokfreqs1[token] * math.log(N/(doc_freqs.get(token, 0)+1)) \n",
    "                    for token in tokfreqs1] if doc_freqs else None\n",
    "        weights2 = [tokfreqs2[token] * math.log(N/(doc_freqs.get(token, 0)+1)) \n",
    "                    for token in tokfreqs2] if doc_freqs else None\n",
    "                \n",
    "        embedding1 = np.average([model[token] for token in tokfreqs1], axis=0, weights=weights1).reshape(1, -1)\n",
    "        embedding2 = np.average([model[token] for token in tokfreqs2], axis=0, weights=weights2).reshape(1, -1)\n",
    "\n",
    "        sim = cosine_similarity(embedding1, embedding2)[0][0]\n",
    "        sims.append(sim)\n",
    "\n",
    "    return sims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Mover's Distance\n",
    "\n",
    "Word mover's distance is a popular alternative to the simple average embedding similarity. The Word Mover's Distance uses the word embeddings of the words in two texts to measure the minimum amount that the words in one text need to \"travel\" in semantic space to reach the words of the other text. Word mover's distance is available in the popular Gensim library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_wmd_benchmark(sentences1, sentences2, model, use_stoplist=False):\n",
    "    \n",
    "    sims = []\n",
    "    for (sent1, sent2) in zip(sentences1, sentences2):\n",
    "    \n",
    "        tokens1 = sent1.tokens_without_stop if use_stoplist else sent1.tokens\n",
    "        tokens2 = sent2.tokens_without_stop if use_stoplist else sent2.tokens\n",
    "        \n",
    "        tokens1 = [token for token in tokens1 if token in model]\n",
    "        tokens2 = [token for token in tokens2 if token in model]\n",
    "        \n",
    "        if len(tokens1) == 0 or len(tokens2) == 0:\n",
    "            tokens1 = [token for token in sent1.tokens if token in model]\n",
    "            tokens2 = [token for token in sent2.tokens if token in model]\n",
    "            \n",
    "        sims.append(-model.wmdistance(tokens1, tokens2))\n",
    "        \n",
    "    return sims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Smooth Inverse Frequency\n",
    "\n",
    "Taking the average of the word embeddings in a sentence, like we did above, is a very crude method of computing sentence embeddings. Most importantly, this gives far too much weight to words that are quite irrelevant, semantically speaking. Smooth Inverse Frequency tries to solve this problem. \n",
    "\n",
    "To compute SIF sentence embeddings, we first compute a weighted average of the token embeddings in the sentence. This procedure is very similar to the weighted average we used above, with the single difference that the word embeddings are weighted by `a/a+p(w)`, where `w` is a parameter that is set to `0.001` by default, and `p(w)` is the estimated relative frequency of a word in a reference corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to perform common component removal: we compute the principal component of the sentence embeddings we obtained above and subtract from them their projections on this first principal component. This corrects for the influence of high-frequency words that mostly have a syntactic or discourse function, such as \"just\", \"there\", \"but\", etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "def remove_first_principal_component(X):\n",
    "    svd = TruncatedSVD(n_components=1, n_iter=7, random_state=0)\n",
    "    svd.fit(X)\n",
    "    pc = svd.components_\n",
    "    XX = X - X.dot(pc.transpose()) * pc\n",
    "    return XX\n",
    "\n",
    "\n",
    "def run_sif_benchmark(sentences1, sentences2, model, freqs={}, use_stoplist=False, a=0.001): \n",
    "    total_freq = sum(freqs.values())\n",
    "    \n",
    "    embeddings = []\n",
    "    \n",
    "    # SIF requires us to first collect all sentence embeddings and then perform \n",
    "    # common component analysis.\n",
    "    for (sent1, sent2) in zip(sentences1, sentences2): \n",
    "        \n",
    "        tokens1 = sent1.tokens_without_stop if use_stoplist else sent1.tokens\n",
    "        tokens2 = sent2.tokens_without_stop if use_stoplist else sent2.tokens\n",
    "        \n",
    "        tokens1 = [token for token in tokens1 if token in model]\n",
    "        tokens2 = [token for token in tokens2 if token in model]\n",
    "        \n",
    "        weights1 = [a/(a+freqs.get(token,0)/total_freq) for token in tokens1]\n",
    "        weights2 = [a/(a+freqs.get(token,0)/total_freq) for token in tokens2]\n",
    "        \n",
    "        embedding1 = np.average([model[token] for token in tokens1], axis=0, weights=weights1)\n",
    "        embedding2 = np.average([model[token] for token in tokens2], axis=0, weights=weights2)\n",
    "        \n",
    "        embeddings.append(embedding1)\n",
    "        embeddings.append(embedding2)\n",
    "        \n",
    "    embeddings = remove_first_principal_component(np.array(embeddings))\n",
    "    sims = [cosine_similarity(embeddings[idx*2].reshape(1, -1), \n",
    "                              embeddings[idx*2+1].reshape(1, -1))[0][0] \n",
    "            for idx in range(int(len(embeddings)/2))]\n",
    "\n",
    "    return sims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The methods above share two important characteristics: \n",
    "\n",
    "- As simple bag-of-word methods, they do take not word order into account.\n",
    "- The word embeddings they use have been learned in an unsupervised manner. \n",
    "\n",
    "Both these characteristics are potential downsides: \n",
    "\n",
    "- Since differences in word order can point to differences in meaning (compare `the dog bites the man` with `the man bites the dog`), we'd like our sentence embeddings to be sensitive to this variation.\n",
    "- Supervised training can help sentence embeddings learn the meaning of a sentence more directly.\n",
    "\n",
    "We can achieve both points by using a pre-trained sentence encoder to produce our sentence embeddings. Several such encoders are available. We'll investigate InferSent and the Google Sentence Encoder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### InferSent\n",
    "\n",
    "[InferSent](https://github.com/facebookresearch/InferSent) is a pre-trained encoder that produces sentence embeddings. \n",
    "More particularly, it is a BiLSTM with max pooling that was trained on the SNLI dataset, 570k English sentence pairs labelled with one of three categories: entailment, contradiction or neutral. InferSent was developed and trained by Facebook Research.\n",
    "\n",
    "Let's first download the resources we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File 'models.py' already there; not retrieving.\n",
      "\n",
      "File 'infersent.allnli.pickle' already there; not retrieving.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://raw.githubusercontent.com/facebookresearch/SentEval/master/examples/models.py\n",
    "!wget -nc https://s3.amazonaws.com/senteval/infersent/infersent.allnli.pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we load the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/torch/serialization.py:316: SourceChangeWarning: source code of class 'models.BLSTMEncoder' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.\n",
      "  warnings.warn(msg, SourceChangeWarning)\n",
      "/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/torch/serialization.py:316: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.LSTM' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.\n",
      "  warnings.warn(msg, SourceChangeWarning)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "infersent = torch.load('infersent.allnli.pickle', map_location=lambda storage, loc: storage)\n",
    "infersent.use_cuda = False\n",
    "infersent.set_glove_path(PATH_TO_GLOVE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can run the benchmark by having InferSent encode the two sets of sentences and compute the cosine similarity between the corresponding sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_inf_benchmark(sentences1, sentences2):\n",
    "    \n",
    "    raw_sentences1 = [sent1.raw for sent1 in sentences1]\n",
    "    raw_sentences2 = [sent2.raw for sent2 in sentences2]\n",
    "    \n",
    "    infersent.build_vocab(raw_sentences1 + raw_sentences2, tokenize=True)\n",
    "    embeddings1 = infersent.encode(raw_sentences1, tokenize=True)\n",
    "    embeddings2 = infersent.encode(raw_sentences2, tokenize=True)\n",
    "    \n",
    "    inf_sims = []\n",
    "    for (emb1, emb2) in zip(embeddings1, embeddings2): \n",
    "        sim = cosine_similarity(emb1.reshape(1, -1), emb2.reshape(1, -1))[0][0]\n",
    "        inf_sims.append(sim)\n",
    "\n",
    "    return inf_sims   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Sentence Encoder\n",
    "\n",
    "The [Google Sentence Encoder](https://www.tensorflow.org/hub/modules/google/universal-sentence-encoder/1) is Google's answer to Facebook's InferSent. It comes in two forms: \n",
    "\n",
    "- a Transformer model that takes the element-wise sum of the context-aware word representations produced by the encoding subgraph of a Transformer model.\n",
    "- a Deep Averaging Network (DAN) where input embeddings for words and bigrams are averaged together and passed through a feed-forward deep neural network.\n",
    "\n",
    "The Transformer model tends to give better results, but at the time of writing, only the DAN-based encoder was available.\n",
    "\n",
    "In contrast to InferSent, the Google Sentence Encoder was trained on a combination of unsupervised data (in a skip-thought-like task) and supervised data (the SNLI corpus).\n",
    "\n",
    "The Google Sentence Encoder can be loaded from the Tensorflow Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "embed = hub.Module(\"https://tfhub.dev/google/universal-sentence-encoder/1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like InferSent above, we'll have the it encode the two sets of sentences and return the similarities between the embeddings it produced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_gse_benchmark(sentences1, sentences2):\n",
    "    sts_input1 = tf.placeholder(tf.string, shape=(None))\n",
    "    sts_input2 = tf.placeholder(tf.string, shape=(None))\n",
    "\n",
    "    sts_encode1 = tf.nn.l2_normalize(embed(sts_input1))\n",
    "    sts_encode2 = tf.nn.l2_normalize(embed(sts_input2))\n",
    "        \n",
    "    sim_scores = tf.reduce_sum(tf.multiply(sts_encode1, sts_encode2), axis=1)\n",
    "    \n",
    "    with tf.Session() as session:\n",
    "        session.run(tf.global_variables_initializer())\n",
    "        session.run(tf.tables_initializer())\n",
    "      \n",
    "        [gse_sims] = session.run(\n",
    "            [sim_scores],\n",
    "            feed_dict={\n",
    "                sts_input1: [sent1.raw for sent1 in sentences1],\n",
    "                sts_input2: [sent2.raw for sent2 in sentences2]\n",
    "            })\n",
    "    return gse_sims\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments\n",
    "\n",
    "Finally, it's time to run the actual experiments. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_experiment(df, benchmarks): \n",
    "    \n",
    "    sentences1 = [Sentence(s) for s in df['sent_1']]\n",
    "    sentences2 = [Sentence(s) for s in df['sent_2']]\n",
    "    \n",
    "    pearson_cors, spearman_cors = [], []\n",
    "    for label, method in benchmarks:\n",
    "        sims = method(sentences1, sentences2)\n",
    "        pearson_correlation = scipy.stats.pearsonr(sims, df['sim'])[0]\n",
    "        print(label, pearson_correlation)\n",
    "        pearson_cors.append(pearson_correlation)\n",
    "        spearman_correlation = scipy.stats.spearmanr(sims, df['sim'])[0]\n",
    "        spearman_cors.append(spearman_correlation)\n",
    "        \n",
    "    return pearson_cors, spearman_cors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AVG-W2V 0.7165220803244023\n",
      "AVG-W2V-STOP 0.7014898380169976\n",
      "AVG-W2V-TFIDF 0.7096471932652145\n",
      "AVG-W2V-TFIDF-STOP 0.6994376660669076\n",
      "AVG-GLOVE 0.6591855208386302\n",
      "AVG-GLOVE-STOP 0.6816024990195564\n",
      "AVG-GLOVE-TFIDF 0.6548097066581027\n",
      "AVG-GLOVE-TFIDF-STOP 0.6552199099252713\n",
      "WMD-W2V 0.6482696802123714\n",
      "WMD-W2V-STOP 0.6392328380282691\n",
      "WMD-GLOVE 0.6249165627113568\n",
      "WMD-GLOVE-STOP 0.6444550832020227\n",
      "SIF-W2V 0.7252383011188146\n",
      "SIF-GLOVE 0.6991998920028533\n",
      "Found 1121(/1122) words with glove vectors\n",
      "Vocab size : 1121\n",
      "INF 0.7008594698958657\n",
      "GSE 0.7282557455668891\n",
      "AVG-W2V 0.7296376389069836\n",
      "AVG-W2V-STOP 0.7140806199702359\n",
      "AVG-W2V-TFIDF 0.6928386300519528\n",
      "AVG-W2V-TFIDF-STOP 0.6868900348081444\n",
      "AVG-GLOVE 0.6860811413331513\n",
      "AVG-GLOVE-STOP 0.7025100460660817\n",
      "AVG-GLOVE-TFIDF 0.6556094094108746\n",
      "AVG-GLOVE-TFIDF-STOP 0.6593063256353848\n",
      "WMD-W2V 0.6417044483790421\n",
      "WMD-W2V-STOP 0.6352091200435409\n",
      "WMD-GLOVE 0.6157933918005197\n",
      "WMD-GLOVE-STOP 0.6373786004788204\n",
      "SIF-W2V 0.729060153111828\n",
      "SIF-GLOVE 0.7210510449681963\n",
      "Found 2265(/2270) words with glove vectors\n",
      "Vocab size : 2265\n",
      "INF 0.7433105062742708\n",
      "GSE 0.7420298313570567\n",
      "AVG-W2V 0.7143924952501431\n",
      "AVG-W2V-STOP 0.7511280387727559\n",
      "AVG-W2V-TFIDF 0.7440437747437346\n",
      "AVG-W2V-TFIDF-STOP 0.7444908937479863\n",
      "AVG-GLOVE 0.5643609008083081\n",
      "AVG-GLOVE-STOP 0.6965468377313421\n",
      "AVG-GLOVE-TFIDF 0.4925152520027595\n",
      "AVG-GLOVE-TFIDF-STOP 0.5093975775124\n",
      "WMD-W2V 0.6634067598190595\n",
      "WMD-W2V-STOP 0.7219439825751874\n",
      "WMD-GLOVE 0.6245676489054579\n",
      "WMD-GLOVE-STOP 0.7125885370488946\n",
      "SIF-W2V 0.7675351521219718\n",
      "SIF-GLOVE 0.7597512132231911\n",
      "Found 7013(/7208) words with glove vectors\n",
      "Vocab size : 7013\n",
      "INF 0.7521106590538243\n",
      "GSE 0.7513780002775707\n",
      "AVG-W2V 0.6245059257341046\n",
      "AVG-W2V-STOP 0.6094722339716129\n",
      "AVG-W2V-TFIDF 0.639912741086196\n",
      "AVG-W2V-TFIDF-STOP 0.5876207802212504\n",
      "AVG-GLOVE 0.4494111229090653\n",
      "AVG-GLOVE-STOP 0.6171554567035256\n",
      "AVG-GLOVE-TFIDF 0.35650444440325635\n",
      "AVG-GLOVE-TFIDF-STOP 0.3618282211628285\n",
      "WMD-W2V 0.5567163068108677\n",
      "WMD-W2V-STOP 0.6562304249920504\n",
      "WMD-GLOVE 0.4790103898566876\n",
      "WMD-GLOVE-STOP 0.6203395263248458\n",
      "SIF-W2V 0.6897598844003188\n",
      "SIF-GLOVE 0.6813510769685793\n",
      "Found 5223(/5343) words with glove vectors\n",
      "Vocab size : 5223\n"
     ]
    }
   ],
   "source": [
    "import functools as ft\n",
    "\n",
    "benchmarks = [(\"AVG-W2V\", ft.partial(run_avg_benchmark, model=word2vec, use_stoplist=False)),\n",
    "              (\"AVG-W2V-STOP\", ft.partial(run_avg_benchmark, model=word2vec, use_stoplist=True)),\n",
    "              (\"AVG-W2V-TFIDF\", ft.partial(run_avg_benchmark, model=word2vec, use_stoplist=False, doc_freqs=doc_frequencies)),\n",
    "              (\"AVG-W2V-TFIDF-STOP\", ft.partial(run_avg_benchmark, model=word2vec, use_stoplist=True, doc_freqs=doc_frequencies)),\n",
    "              (\"AVG-GLOVE\", ft.partial(run_avg_benchmark, model=glove, use_stoplist=False)),\n",
    "              (\"AVG-GLOVE-STOP\", ft.partial(run_avg_benchmark, model=glove, use_stoplist=True)),\n",
    "              (\"AVG-GLOVE-TFIDF\", ft.partial(run_avg_benchmark, model=glove, use_stoplist=False, doc_freqs=doc_frequencies)),\n",
    "              (\"AVG-GLOVE-TFIDF-STOP\", ft.partial(run_avg_benchmark, model=glove, use_stoplist=True, doc_freqs=doc_frequencies)),\n",
    "              (\"WMD-W2V\", ft.partial(run_wmd_benchmark, model=word2vec, use_stoplist=False)), \n",
    "              (\"WMD-W2V-STOP\", ft.partial(run_wmd_benchmark, model=word2vec, use_stoplist=True)), \n",
    "              (\"WMD-GLOVE\", ft.partial(run_wmd_benchmark, model=glove, use_stoplist=False)), \n",
    "              (\"WMD-GLOVE-STOP\", ft.partial(run_wmd_benchmark, model=glove, use_stoplist=True)), \n",
    "              (\"SIF-W2V\", ft.partial(run_sif_benchmark, freqs=frequencies, model=word2vec, use_stoplist=False)),\n",
    "              (\"SIF-GLOVE\", ft.partial(run_sif_benchmark, freqs=frequencies, model=glove, use_stoplist=False)), \n",
    "              (\"INF\", run_inf_benchmark),\n",
    "              (\"GSE\", run_gse_benchmark)]\n",
    "\n",
    "pearson_results, spearman_results = {}, {}\n",
    "pearson_results[\"SICK-DEV\"], spearman_results[\"SICK-DEV\"] = run_experiment(sick_dev, benchmarks)\n",
    "pearson_results[\"SICK-TEST\"], spearman_results[\"SICK-TEST\"] = run_experiment(sick_test, benchmarks)\n",
    "pearson_results[\"STS-DEV\"], spearman_results[\"STS-DEV\"] = run_experiment(sts_dev, benchmarks)\n",
    "pearson_results[\"STS-TEST\"], spearman_results[\"STS-TEST\"] = run_experiment(sts_test, benchmarks)  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "Let's take a look at our results. We'll mostly work with Pearson correlation, as is standard in the literature, except where Spearman correlation sheds additional light on our findings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = (10,5)\n",
    "\n",
    "pearson_results_df = pd.DataFrame(pearson_results)\n",
    "pearson_results_df = pearson_results_df.transpose()\n",
    "pearson_results_df = pearson_results_df.rename(columns={i:b[0] for i, b in enumerate(benchmarks)})\n",
    "\n",
    "spearman_results_df = pd.DataFrame(spearman_results)\n",
    "spearman_results_df = spearman_results_df.transpose()\n",
    "spearman_results_df = spearman_results_df.rename(columns={i:b[0] for i, b in enumerate(benchmarks)})\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baselines\n",
    "\n",
    "- Simple word2vec embeddings outperform GloVe embeddings.\n",
    "- With word2vec, it's unclear whether using a stoplist or tf-idf weighting helps. With STS it sometimes does; with SICK it does not. Simply computing an unweighted average of all word2vec embeddings consistently performs pretty well.\n",
    "- With GloVe, using a stoplist looks like a very good idea. Using tf-idf weights does not help, with or without a stoplist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_results_df[[b[0] for b in benchmarks if b[0].startswith(\"AVG\")]].plot(kind=\"bar\").legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Mover's Distance\n",
    "\n",
    "Based on our results, there's little reason to use Word Mover's Distance rather than simple word2vec averages. Only on STS-TEST, and only in combination with a stoplist, does WMD clearly leave the baselines behind."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_results_df[[\"AVG-W2V\", \"WMD-W2V\", \"WMD-W2V-STOP\", \"AVG-GLOVE-STOP\", \"WMD-GLOVE\", \"WMD-GLOVE-STOP\"]].plot(kind=\"bar\").legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Smooth Inverse Frequency\n",
    "\n",
    "Smooth Inverse Frequency is the most consistent performer in our tests. On the SICK data, it does about as well as its baseline competitors, on STS it outranks them by a clear margin. Note there is little difference between SIF with word2vec embeddings and SIF with GloVe embeddings. This is remarkable, given the large differences we observed above. It shows SIF's weighting and common component removal is a very effective alternative to using a stoplist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_results_df[[\"AVG-W2V\", \"AVG-GLOVE-STOP\", \"SIF-W2V\", \"SIF-GLOVE\"]].plot(kind=\"bar\").legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretrained encoders\n",
    "\n",
    "Pre-trained encoders have a lot to be said for them. However, our results indicate they are not yet able to capitalize fully on their training regime. Throughout our tests, Google's Sentence Encoder looks like a better choice than InferSent. However, the Pearson correlation coefficient shows very little difference with Smooth Inverse Frequency. The differences in Spearman correlation are more outspoken. This may indicate that the Google Sentence Encoder more often gets the relative ordering of the sentences right, but not necessarily the relative differences between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_results_df[[\"SIF-W2V\", \"INF\", \"GSE\"]].plot(kind=\"bar\").legend(loc=\"lower left\")\n",
    "spearman_results_df[[\"SIF-W2V\", \"INF\", \"GSE\"]].plot(kind=\"bar\").legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = (20,13)\n",
    "pearson_results_df.plot(kind=\"bar\").legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "These are the most important conclusions:\n",
    "\n",
    "- When you're computing sentence similarity, word2vec embeddings are a safer choice than GloVe embeddings.\n",
    "- Although an unweighted average of the word embeddings in the sentence holds its own as a simple baseline, Smooth Inverse Frequency is usually a stronger alternative.\n",
    "- When you can use a pre-trained encoder, pick Google's Sentence Encoder, but remember its performance gain may not be all that spectacular."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}