notebooks/cnn/cnn_torch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Дисклеймер\n",
    "Эту тетрадку нужно запускать в колабе или в vast.ai. Не мучатесь с установкой библиотек и с обучением на cpu.\n",
    "\n",
    "Я разбирался в торче пока делал этот семинар, так что относитесь к моим наработкам как к обучающему материалу, а не стандарту. Скорее всего, есть более правильные, эффективные, красивые решения. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting torch\n",
      "  Downloading torch-1.8.1-cp36-cp36m-manylinux1_x86_64.whl (804.1 MB)\n",
      "\u001b[K     |█████████████▋                  | 342.3 MB 79.2 MB/s eta 0:00:066"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub data rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_data_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |██████████████████████████▍     | 663.3 MB 84.6 MB/s eta 0:00:02"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub data rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_data_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 804.1 MB 113.4 MB/s eta 0:00:01\u001b[K     |████████████████████████████████| 804.1 MB 34 kB/s \n",
      "\u001b[?25hCollecting torch-vision\n",
      "  Downloading torch_vision-0.1.6.dev0-py2.py3-none-any.whl (23 kB)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch) (3.7.4.3)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.19.5)\n",
      "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from torch) (0.8)\n",
      "Installing collected packages: torch, torch-vision\n",
      "Successfully installed torch-1.8.1 torch-vision-0.1.6.dev0\n",
      "\u001b[33mWARNING: You are using pip version 20.2.4; however, version 21.1.1 is available.\n",
      "You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install torch torch-vision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "unzip is already the newest version (6.0-21ubuntu1.1).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.\n"
     ]
    }
   ],
   "source": [
    "!apt-get install unzip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  data.zip\n",
      "  inflating: lenta_sample.csv        \n",
      "  inflating: __MACOSX/._lenta_sample.csv  \n"
     ]
    }
   ],
   "source": [
    "!unzip data.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 9.5 MB 3.8 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scikit-learn\n",
      "  Downloading scikit_learn-0.24.2-cp36-cp36m-manylinux2010_x86_64.whl (22.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 22.2 MB 51.2 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting matplotlib\n",
      "  Downloading matplotlib-3.3.4-cp36-cp36m-manylinux1_x86_64.whl (11.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 11.5 MB 58.7 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.19.5)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas) (2021.1)\n",
      "Collecting threadpoolctl>=2.0.0\n",
      "  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\n",
      "Collecting joblib>=0.11\n",
      "  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)\n",
      "\u001b[K     |████████████████████████████████| 303 kB 58.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scipy>=0.19.1\n",
      "  Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 25.9 MB 50.9 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting cycler>=0.10\n",
      "  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)\n",
      "Collecting pillow>=6.2.0\n",
      "  Downloading Pillow-8.2.0-cp36-cp36m-manylinux1_x86_64.whl (3.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 3.0 MB 49.3 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting kiwisolver>=1.0.1\n",
      "  Downloading kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 58.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.4.7)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
      "Installing collected packages: pandas, threadpoolctl, joblib, scipy, scikit-learn, cycler, pillow, kiwisolver, matplotlib\n",
      "Successfully installed cycler-0.10.0 joblib-1.0.1 kiwisolver-1.3.1 matplotlib-3.3.4 pandas-1.1.5 pillow-8.2.0 scikit-learn-0.24.2 scipy-1.5.4 threadpoolctl-2.1.0\n",
      "\u001b[33mWARNING: You are using pip version 20.2.4; however, version 21.1.1 is available.\n",
      "You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas scikit-learn matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Сверточные нейронные сети"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Несколько полносвязных слоев (с нелинейностями посередине) - это уже полноценная нейронная сеть. Однако это скорее какая-то базовая вещь. Действительно нужен полносвязный слой только в конце сети, а в качестве основных используются сверточные или рекуррентные слои. Главное их отличие (=преимущество) - учитывание структукры данных. В этом семинаре мы попробуем использовать сверточные сети. Лучше всего они работают в картинках, но для текстов их тоже можно применять (они быстрее рекуррентных и могут лучше работать на длинных последовательностях).\n",
    "\n",
    "Про то как работают сверточные слои посмотрите в этом видео - https://www.youtube.com/watch?v=bNb2fEVKeEo\n",
    "\n",
    "Основная идея - использовать много небольших (сильно меньше размера текста или картинки) матриц, которые применяются к кусочкам векторизованного текста/картинки. Между кусочком и матрицей считается близость, и эта близость передается дальше по сети. Т.е. кусочек как-бы **сворачивается** в одно число. \n",
    "\n",
    "Эти небольшие матрицы обучаются на данных и, например, для картинок в них собираются различные паттерны, важные для задачи. Например, для определения рукописных цифр, такие матрицы могут представлять горизонтальные черточки, волнистые линии, кружки и т.д. Есть даже целое направление исследований, которое занимается визуализацией и интерпретацией обученных сверток.\n",
    "\n",
    "Из-за того, что матрицы применяются к кусочкам, а не отедльным элементам, гораздо лучше учитывается структура. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Классификация с помощью CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте посмотрим, как в торче использовать сверточные сети и как их соединять с Embedding и Linear слоями."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.utils.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# данные - статьи из ленты ру, нужно предсказать категорию новости\n",
    "data = pd.read_csv('lenta_sample.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dropna(subset=['topic', 'text'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка нам тут особо не важна."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    tokens = text.lower().split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В предыдущей домашке не хватало фмеры. Ее нет в керасе, но можно взять вот такую функцию (взятую со стаковерфлоу)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка точно такая же"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# словарь\n",
    "vocab = Counter()\n",
    "\n",
    "for text in data.text:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "670782"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# отфильтрованный словарь\n",
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 30:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55293"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# индексируем слова\n",
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обертка для данных - такая же как и в предыдущем семинаре"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Dataset(torch.utils.data.Dataset):\n",
    "\n",
    "    def __init__(self, word2id, MAX_LEN, texts, targets):\n",
    "        \n",
    "        # храним сырой датасет\n",
    "        # еще лучше можно сделать, если считывать из файла при генерации\n",
    "        self.texts = [torch.LongTensor([word2id[w] for w in self.preprocess(t) if w in word2id][:MAX_LEN]) for t in texts]\n",
    "        self.texts = torch.nn.utils.rnn.pad_sequence(self.texts, batch_first=True)\n",
    "        \n",
    "        # мы не считаем словарь прямо тут, т.к. нам нужно шерить словарь между трейном и валидацией\n",
    "        self.word2id = word2id\n",
    "        self.MAX_LEN = MAX_LEN\n",
    "        self.length = len(texts)\n",
    "        # таргеты можно сразу целиком выделить \n",
    "        \n",
    "        self.target = torch.LongTensor(targets)\n",
    "\n",
    "    def __len__(self):\n",
    "        # нужно уметь возвращать длину датасета, в нашем случае это просто сделать\n",
    "        # если не загружать данные целиком, а читать из файла - это может быть сложнее\n",
    "        return self.length\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "\n",
    "        # в index будут индексы от 0 до self.length\n",
    "        # по ним мы достанем тексты, предобработаем,переведем в векторы, западим и вернем\n",
    "\n",
    "        ids = self.texts[index]\n",
    "        y = self.target[index]\n",
    "\n",
    "        return ids, y\n",
    "    \n",
    "    def preprocess(self, text):\n",
    "        tokens = text.lower().split()\n",
    "        tokens = [token.strip(punctuation) for token in tokens]\n",
    "        tokens = [token for token in tokens if token]\n",
    "        return tokens\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = data.text.values\n",
    "id2label = {i:l for i,l in enumerate(set(data.topic))}\n",
    "label2id = {l:i for i,l in id2label.items()}\n",
    "targets = [label2id[l] for l in data.topic]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_texts, valid_texts, train_targets, valid_targets = train_test_split(texts, targets, test_size=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_set = Dataset(word2id, MAX_LEN, train_texts, train_targets)\n",
    "training_generator = torch.utils.data.DataLoader(training_set, batch_size=256, shuffle=True, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_set = Dataset(word2id, MAX_LEN, valid_texts, valid_targets)\n",
    "valid_generator = torch.utils.data.DataLoader(valid_set, batch_size=256, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для сравнения сначала обучим сеть только с помощью Dense слоев. Трейн луп такой же как и в предыдущем семинаре"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def train(model, iterator, optimizer, criterion, print_every=1000):\n",
    "    \n",
    "    epoch_loss = []\n",
    "    epoch_f1 = []\n",
    "    \n",
    "    model.train()  \n",
    "\n",
    "    for i, (texts, ys) in enumerate(iterator):\n",
    "        \n",
    "        optimizer.zero_grad()   \n",
    "        predictions = model(texts.to(device)).squeeze()  \n",
    "        loss = criterion(predictions, ys.to(device))        \n",
    "        \n",
    "        \n",
    "        \n",
    "        loss.backward()\n",
    "        optimizer.step()     \n",
    "        preds = predictions.detach().to('cpu').numpy().argmax(1).tolist()\n",
    "        y_true = ys.tolist()\n",
    "        \n",
    "        epoch_loss.append(loss.item())\n",
    "        epoch_f1.append(f1_score(y_true, preds, average=\"micro\"))\n",
    "        \n",
    "        if not (i+1) % print_every:\n",
    "            print(f'Loss: {np.mean(epoch_loss)}; F1 measure: {np.mean(epoch_f1)}')\n",
    "        \n",
    "    return np.mean(epoch_f1)\n",
    "\n",
    "\n",
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = []\n",
    "    epoch_f1 = []\n",
    "    \n",
    "    model.eval()  \n",
    "    with torch.no_grad():\n",
    "        for texts, ys in iterator:   \n",
    "\n",
    "            predictions = model(texts.to(device)).squeeze()  \n",
    "            loss = criterion(predictions, ys.to(device))        \n",
    "            preds = predictions.detach().to('cpu').numpy().argmax(1).tolist()\n",
    "            y_true = ys.tolist()\n",
    "            \n",
    "\n",
    "            epoch_loss.append(loss.item())  \n",
    "            epoch_f1.append(f1_score(y_true, preds, average=\"micro\"))\n",
    "            \n",
    "    return np.mean(epoch_f1)# / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# сеть с двумя полносвязными слоями и активацией\n",
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, hidden, output_dim):\n",
    "        super().__init__()          \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.fc = nn.Linear(embedding_dim, hidden)\n",
    "        self.fc_1 = nn.Linear(hidden, output_dim)\n",
    "        self.act = nn.LogSoftmax(1) \n",
    "        self.relu = nn.ReLU()       \n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text)   \n",
    "        hidden = torch.mean(embedded, dim=1)\n",
    "        hidden = self.relu(self.fc(hidden))\n",
    "        dense_outputs=self.fc_1(hidden) \n",
    "\n",
    "        outputs=self.act(dense_outputs)\n",
    "        \n",
    "        return outputs\n",
    "\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(word2id), 30, 30, len(label2id))\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss() \n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Запускаем обучение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 2.6197411823272705; F1 measure: 0.1808203125\n",
      "Loss: 2.4586977422237397; F1 measure: 0.19806640625\n",
      "Loss: 2.373960111141205; F1 measure: 0.20907552083333333\n",
      "Loss: 2.293777464926243; F1 measure: 0.235419921875\n",
      "Loss: 2.1999969239234924; F1 measure: 0.2706328125\n",
      "Eval -  0.46314655172413793\n",
      "1\n",
      "Loss: 1.5232779133319854; F1 measure: 0.5215234375\n",
      "Loss: 1.4571399873495101; F1 measure: 0.54337890625\n",
      "Loss: 1.4014333546161652; F1 measure: 0.5609895833333334\n",
      "Loss: 1.355219143331051; F1 measure: 0.57517578125\n",
      "Loss: 1.313402940273285; F1 measure: 0.588875\n",
      "Eval -  0.6512631704980844\n",
      "2\n",
      "Loss: 1.0344296985864638; F1 measure: 0.6799609375\n",
      "Loss: 1.0047901949286462; F1 measure: 0.68927734375\n",
      "Loss: 0.9862580219904582; F1 measure: 0.6943098958333334\n",
      "Loss: 0.9652436164021492; F1 measure: 0.701083984375\n",
      "Loss: 0.9504220184087754; F1 measure: 0.705734375\n",
      "Eval -  0.7233494971264368\n",
      "3\n",
      "Loss: 0.8189972084760666; F1 measure: 0.74671875\n",
      "Loss: 0.812115375995636; F1 measure: 0.7473046875\n",
      "Loss: 0.8023925479253133; F1 measure: 0.7495442708333333\n",
      "Loss: 0.7957931512594223; F1 measure: 0.752353515625\n",
      "Loss: 0.7878573575019836; F1 measure: 0.7547265625\n",
      "Eval -  0.7482405411877395\n",
      "4\n",
      "Loss: 0.7163863027095795; F1 measure: 0.7746484375\n",
      "Loss: 0.7063124039769173; F1 measure: 0.77802734375\n",
      "Loss: 0.70201800207297; F1 measure: 0.7785416666666667\n",
      "Loss: 0.6974907685816288; F1 measure: 0.780703125\n",
      "Loss: 0.6953367521762848; F1 measure: 0.7814453125\n",
      "Eval -  0.7612601772030652\n",
      "5\n",
      "Loss: 0.6381947246193885; F1 measure: 0.801328125\n",
      "Loss: 0.6351038740575313; F1 measure: 0.8006640625\n",
      "Loss: 0.6329314880569776; F1 measure: 0.8010026041666667\n",
      "Loss: 0.6318752785772085; F1 measure: 0.80125\n",
      "Loss: 0.6284054705500602; F1 measure: 0.801921875\n",
      "Eval -  0.7726394875478927\n",
      "6\n",
      "Loss: 0.583585424721241; F1 measure: 0.8152734375\n",
      "Loss: 0.5796784365177154; F1 measure: 0.81775390625\n",
      "Loss: 0.5782570865750313; F1 measure: 0.8176432291666667\n",
      "Loss: 0.5759222892671824; F1 measure: 0.81875\n",
      "Loss: 0.5757328113317489; F1 measure: 0.8190625\n",
      "Eval -  0.7789146312260536\n",
      "7\n",
      "Loss: 0.5341010338068009; F1 measure: 0.831484375\n",
      "Loss: 0.5356463412940502; F1 measure: 0.8300390625\n",
      "Loss: 0.5386197704076767; F1 measure: 0.8299479166666667\n",
      "Loss: 0.5349517285823822; F1 measure: 0.83123046875\n",
      "Loss: 0.5324598643779754; F1 measure: 0.832328125\n",
      "Eval -  0.7886871408045977\n",
      "8\n",
      "Loss: 0.48952979385852813; F1 measure: 0.8453515625\n",
      "Loss: 0.4932489763200283; F1 measure: 0.8452734375\n",
      "Loss: 0.4943637558817863; F1 measure: 0.8448307291666667\n",
      "Loss: 0.49427898682653904; F1 measure: 0.8447265625\n",
      "Loss: 0.4922732102274895; F1 measure: 0.8455859375\n",
      "Eval -  0.7920222701149425\n",
      "9\n",
      "Loss: 0.46336703181266786; F1 measure: 0.85640625\n",
      "Loss: 0.46282783165574076; F1 measure: 0.85646484375\n",
      "Loss: 0.4612191939353943; F1 measure: 0.8561067708333333\n",
      "Loss: 0.4582536684721708; F1 measure: 0.8560546875\n",
      "Loss: 0.4556128804087639; F1 measure: 0.856703125\n",
      "Eval -  0.7923473419540229\n"
     ]
    }
   ],
   "source": [
    "f1s = []\n",
    "f1s_eval = []\n",
    "\n",
    "for i in range(10):\n",
    "    print(i)\n",
    "    f1s.append(train(model, training_generator, optimizer, criterion, 100))\n",
    "    evl = evaluate(model, valid_generator, criterion)\n",
    "    print('Eval - ', evl)\n",
    "    f1s_eval.append(evl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvG0lEQVR4nO3deXxV1bn/8c+TeSBzgEASCAgyT4qIRZzHWrXV1qHq7+rtldte7fxra4fbwdv25+3trZ1sq22tHRzaaq3a2sYJUStWQBEM80yAhCSQiczJ8/vjHCAJgQTIyUnO+b5fr7zO2Xuvvc9zjrKevddaey9zd0REJHrFhDsAEREJLyUCEZEop0QgIhLllAhERKKcEoGISJRTIhARiXJKBBKVzOwhM/tmH8tuM7OLjrIt2cyeMbMaM/tj/0Z5xGeNMbN6M4sN5edI9FEiEDk5HwRGAjnu/iEzG2VmT5vZbjNzMyvqrw9y9x3uPszd2/vrmCKgRCByssYCG9y9LbjcAfwduDZ8IXVlZnHhjkEGNyUCGbSCTTKfM7NVZnbAzH5pZiPN7G9mVmdmL5hZVqfyV5lZiZlVm9nLZjal07Y5ZvZWcL/fA0ndPut9ZrYyuO/rZjazD/F9A/gqcH2wyeYj7l7u7j8BlvX3dzSzouBVRlxwOdvMfhW8+thvZn8Orj/PzErN7AtmVgb8yswSzez7wbK7g+8T+xKjRD4lAhnsrgUuBk4FrgT+BnwJGE7g/99PAJjZqcCjwKeC254FnjGzBDNLAP4M/BbIBv5IpzN2M5sDPAj8O5AD3A883VtF6e5fA74N/D7YZPPLUH7HHvwWSAGmASOAezttyyPwXccCi4AvA/OB2cAsYB7wlROMVyKMEoEMdj8KnmXvAl4F/unub7t7E/AkMCdY7nrgr+7+vLu3At8FkoH3EKgA44Hvu3uruz9O1zP2RcD97v5Pd293918DzcH9BkJfv+MhZjYKuBz4qLvvD36vJZ2KdABfc/dmd28EbgLudve97l4BfAO4JdRfTIYGtR3KYFfe6X1jD8vDgu9HA9sPbnD3DjPbCeQD7cAu7/qExe2d3o8F/sXMPt5pXULwmAOhr9+xs0Jgn7vvP8oxK4KJ5KAuv0/w/UB9PxnkdEUgkWI3gQodADMzApXlLmAPkB9cd9CYTu93At9y98xOfynu/uhABH6CdgLZZpZ5lO3dHyvc5fch8P13hyAuGYKUCCRS/AG4wswuNLN44LMEmndeB5YCbcAnzCzezK4h0EZ+0M+Bj5rZmRaQamZXmFnaiQRiZknAwf6FxOByv3L3PQT6En5iZlnB73XOMXZ5FPiKmQ03s1wCndy/6++4ZGhSIpCI4O7rgZuBHwGVBDpdr3T3FndvAa4BbgX2EehP+FOnfZcDtwM/BvYDm4JlT1QjUB98vy64HAq3AK3Bz9hLoKP8aL4JLAdWAauBt4LrRDBNTCMiEt10RSAiEuWUCEREopwSgYhIlFMiEBGJckPuhrLc3FwvKioKdxgiIkPKihUrKt19eE/bhlwiKCoqYvny5eEOQ0RkSDGz7UfbpqYhEZEop0QgIhLllAhERKLckOsj6ElrayulpaU0NTX1XngIS0pKoqCggPj4+HCHIiIRJCISQWlpKWlpaRQVFdH1AZORw92pqqqitLSUcePGhTscEYkgEdE01NTURE5OTsQmAQAzIycnJ+KvekRk4EVEIgAiOgkcFA3fUUQGXkQ0DYmIRJq6plbKaprYU9N06PWCySOYUZDR75+lRNAPqqureeSRR/iP//iP49rvve99L4888giZmZmhCUxEBh13p7apjT01jV0q+bLg8sF19c1tXfYzg5xhCUoEg1V1dTU/+clPjkgEbW1txMUd/Sd+9tlnQx2aiAwgd6e6oTVQmdceruh3V3ddbmhp77KfGYxISyQvI5kJw4excGIuozKSyMtIDrymJzEyPYmEuNC05isR9IO77rqLzZs3M3v2bOLj40lKSiIrK4t169axYcMG3v/+97Nz506ampr45Cc/yaJFi4DDj8uor6/n8ssv5+yzz+b1118nPz+fp556iuTk5DB/MxE5yN3Zd6Cl01l7tzP62ib21DTS1NrRZb8Yg5HpSeRlJDE5L43zJ40IVvJJjMpIYlRGMsPTEomPDV+XbcQlgm88U8Ka3bX9esypo9P52pXTjrr9nnvu4d1332XlypW8/PLLXHHFFbz77ruHhnk++OCDZGdn09jYyBlnnMG1115LTk5Ol2Ns3LiRRx99lJ///Odcd911PPHEE9x88839+j1E5NhqGlrZvu8A26sa2LGvge1VB9ixryF4Rt9ES1vXSj42xsgLVvJTR6dz0ZQRh8/iM5IYnZFM7rAE4sJYyfdFxCWCwWDevHldxvr/8Ic/5MknnwRg586dbNy48YhEMG7cOGbPng3A6aefzrZt2wYqXJGo0dHhlNU2BSv6QIW/fV8DO/c1sL2qgZrG1i7lc4clMCY7hVmFmVze6Sw+LyOZ0RlJ5AxLJDZm6I/mi7hEcKwz94GSmpp66P3LL7/MCy+8wNKlS0lJSeG8887r8V6AxMTEQ+9jY2NpbAzVfOcika2ptZ3S/YGKvfOZ/fZ9DZTua6Sl/fBZfWyMUZCVzJjsFN43cxRjc1IYk50afE0hNTHiqsgeRce3DLG0tDTq6up63FZTU0NWVhYpKSmsW7eON954Y4CjE4k81Q0th87md1Qd6PS+gbLaridaqQmxjMlJ5dQRaVw8ZSSF2SmMzUlhbHYqozOTBn2zzUBQIugHOTk5LFiwgOnTp5OcnMzIkSMPbbvsssv42c9+xpQpU5g0aRLz588PY6QiQ0P7oSacA+zoVMlv3xdYrm3qOrRyRFoiY7JTeM+EHMYePKMPntXnpCboZsxemLuHO4bjMnfuXO8+Mc3atWuZMmVKmCIaWNH0XSWyNba0s/NQE06gU3ZHsMIv3d+1CSc+1ijIClTsY4Jn9IHXVAqzk0lJ0Dltb8xshbvP7Wmbfj0RCYmDwy0Pnc0H2+sPdtLurWvuUj4tMY4xOSlMHpXGJdPyDlX2Y7JTGJ2ZHBGdsoOVEoGInLC29g52Vzd1GXK5o9NInO53x+alJzEmJ4VzTx3O2JyUYHt9KmOzU8hMiVcTTpgoEYjIMdU3t7Gj03DLg00426sa2FXdSHvH4eblhLgYCrOSGZuTypnjsg8144zNSaEgK4Wk+NgwfhM5GiUCEcHdKd3fyLqyOtaX1bJpb/2hJp2qAy1dymamxDM2O4WZBRlcOWsUY7NTD3XM5qUnEaMmnCFHiUAkytQ0trK+rI51ZbWsK6tj3Z5aNpTXd2nGGZ2RxNicVC6eOpIxwaGWY7IDI3EykjVDXqRRIhCJUC1tHWyprGd9WR1r9wTO9NeX1bG75vA4+/SkOCaPSuea0/KZlJfG5Lx0Th05jLQkVfbRRIkgDIYNG0Z9fX24w5AI4R4Yc79uT13gDD9Y4W+uqKe1PdB+Hx9rnDJ8GPPGZTMpL53JeWlMHpVGXnqSOmhFiUBkKKlvbjvUrBN4DTTtdL7BanRGEpPy0jh/8ohAhZ+Xzrjc1JA9wliGvpAmAjO7DPgBEAv8wt3v6bZ9DPBrIDNY5i53H3IP6b/rrrsoLCzkjjvuAODrX/86cXFxLF68mP3799Pa2so3v/lNrr766jBHKkNFW3sH26oOBCv6QIW/vryWnfsOP4NqWGIck/LSeN+s0UzJS2NSXjqTRqaRkaJmHTk+Ibuz2MxigQ3AxUApsAy40d3XdCrzAPC2u//UzKYCz7p70bGO2+udxX+7C8pW9+M3AfJmwOX3HHXz22+/zac+9SmWLFkCwNSpUykuLiYjI4P09HQqKyuZP38+GzduxMxOqmlIdxZHnoaWNtbsrmX1rhre3VXLurJaNu6tP/TI49gYY3xuarANP3CGPykvjYKsZDXrSJ+F687iecAmd98SDOIx4GpgTacyDqQH32cAu0MYT8jMmTOHvXv3snv3bioqKsjKyiIvL49Pf/rTvPLKK8TExLBr1y7Ky8vJy8sLd7gSRo0t7azZU8vq0mpW76pl9a5qNu2t5+BQ/OFpiUwZlc6CCblMGhloxz9l+DCNv5eQCmUiyAd2dlouBc7sVubrwHNm9nEgFbiopwOZ2SJgEcCYMWOO/anHOHMPpQ996EM8/vjjlJWVcf311/Pwww9TUVHBihUriI+Pp6ioqMfHT0vkamptZ+2ewJn+6tIaVu+qYePe+kM3YOUOS2RmQQaXTx/FjPwMZhRkMDI9KcxRSzQKd2fxjcBD7v6/ZnYW8Fszm+7uXaYBcvcHgAcg0DQUhjh7df3113P77bdTWVnJkiVL+MMf/sCIESOIj49n8eLFbN++PdwhSgg1tbazrqwuWOkHzvY3lNcdqvRzUgOTjl8ydSTT8zOYWZDJyPRENe0MRh3t0NoIbU2Bv9Ym6DjYGe/g3sPr0bYF18NR9utlW/djD58EGQX9/pVDmQh2AYWdlguC6zr7CHAZgLsvNbMkIBfYG8K4QmLatGnU1dWRn5/PqFGjuOmmm7jyyiuZMWMGc+fOZfLkyeEOUfpJc1s76w9V+jWsKq1hQ3kdbcFKPzs1gen5GVw4eQQzCjKYkZ/BqAwN0zwh7a2HK+O2xuDrwQq6sdNrc6ftweVjbe9ynG77dLT2Hle4XPE9OOMj/X7YUCaCZcBEMxtHIAHcAHy4W5kdwIXAQ2Y2BUgCKkIYU0itXn24kzo3N5elS5f2WE73EAwdLW0dbCivY1WwaWf1rmrWl9UdGp+fmRLPjPwMFk0az8yCDKbnZ5CfqU7cXjXXQ3051O2BurJOr2Vdl1sPnPhnxCZAXDLEJUJ8UuD9oddkSMmGuKTA+7jEbtuTAtsObo852EdjYNbzKxx926H/HyxY9Gjbejl21uEpcPtTyBKBu7eZ2Z1AMYGhoQ+6e4mZ3Q0sd/engc8CPzezTxO4RrrVh9oECRIxWtsDlf7qQ5V+Dev21B16Ln56UhwzCzL5yNmBSn9GfoZG7nTX0hCoxI+o5Mu7VvYtPczoF5cEaaMgLQ9GzYRTL4XkrE4VcrfKPC4xWIknHX49VC6pU+UtvQlpH0HwnoBnu637aqf3a4AFoYxB5Gia29pZsX0/r22s5PXNVazZU3toyGZaUhwz8jO47ewiZuRnMDM/k8LsKK70WxsPV+L1ZUc5iy+D5poj941NDFTuaaNg5DSYcOHh5bQ8GJYXeE3K6HR2LAMp3J3F/cbdI/4fqS6WTo67s6G8nlc3VvDqxkre3LqPxtZ24mKM2YWZ3PqeokBHbn4GY7JTIvcpmq2N0FgNTTXQFHztvtxQ1fVMvqn6yOPEJhyu0IdPgvHnBZfzulb0SZmq4Ae5iEgESUlJVFVVkZOTE7HJwN2pqqoiKUnDC4/H3tomXttUyWsbK3ltU+WhWbHGD0/lurkFLJw4nDPHZw+th6x1tENzbe+V+dGW25uPcXAgYVigSSYtD3InwLiFMGzk4Yr94Gtylir4CBERiaCgoIDS0lIqKoZsP3OfJCUlUVDQ/0PHIkljSzv/3Fp1qOJfVxZoi85OTWDBhFwWTshlwcRc8jOTwxxpkDs07IOanVBTGjj7bqwOVt7VXSvzxprA++ZaDg077InFQnJmoKklKfiaUdB1ufP25Mzg+kxISofYIZQUpV9ERCKIj49n3LjQ9KbL4NbR4ZTsruWVjRW8trGSFdv309LeQUJcDGcUZfGFyyazcGIuU0elh6epp7UJancFKvlDfzu7Lrc1HrlffGrXyjq9AEZO76UyD75PSNWZuhyXiEgEEl1K9zfw2sZKXt1UyeubKtnfEBj3PWVUOrcuKOLsCbmcUZRNckKIR424w4HKIyv2zssHerglZlhe4Ax95LTAyJiMwsByRgGkjw40ueisXAaQEoEMenVNrSzdXMVrmyp5dWMlWysDY8tHpidyweSRLJyYy4IJuQxPS+zfD25thJpdx67ou7e3x6ccrtjzZnSt5A9W9HH9HKfISVIikEGnrb2Dd0qreWVDoJ1/5c5q2juclIRY5o/P4Zb5Y1k4MZcJI4ad3OCA5nrYv+3wX/dKvqGy2w4W6CjNKIBRs2DyFUdW9OpAlSFIiUDCzt3ZWnng0Bn/G5urqGtuI8ZgRkEmHzv3FM6emMtpY7KOb3KVjo5A5+uhyn5r14r/QLfBBQnDDlfso+cEK/dOFX3aKIhL6LfvLTJYKBFI2GyvOsAvXt3KS+v2sqs60GFamJ3M+2aN5pyJubznlNzeJ1lpaehauXep7Ld3bbqxmECFnlUEk94beM0qguxxkDlWZ/MStZQIZMBtrqjnvsWbeGrlbuJijPMnjeBj553Cwom5jM1J7VrYPfC4gn1be67w68u7lk9Ig+yiwA1Op17WtbLPKFQnrEgPlAhkwGwor+PHL23imVW7SYqL5V8XFHH7OeMZkeRQvQMqX4WN27pV9tu7Da+0w2f1Ey8OPIQrqyjwmj1OZ/UiJ0CJQEJuze5afrx4I8+uLiM1IZbPzR/GLXnbSdv9FPzytUAS6Cw+NVCp50yACRcdruiziiCzUKNuRPqZEoGEzOrSGn740kbeXrOB8xPX8fTY7UxrfofYt7cGCiRnQ9HZMOeWTmf2RZCaq7N6kQGkRCD97p0NW3j1+SdJ27OUu+LWckpSaWBDdTqMXQBnLoJx58CIqRBzHKOARCQklAjk5DXVwvbX2fPOczRvfJkZLVuYZU5rQhI29j1wyu2BB5flzYJY/S8nMtjoX6Ucv5YG2PkGbH0F3/oq7H4b83ayPZ5VMZOoHP9Rpp99JUljz9C4e5EhQIlAetfWDKXLYOsrsPXVwPuOVjosjg1xp/Jc65WsS5rNmQsv47qzTg39M35EpF8pEciR2lth99vBiv8V2PnPwCTfFoOPmsX2U2/loT2F/KG8gMyMTD52xSl8b24hSfFKACJDkRKBBCY6KVsdqPS3vQrbX4eW+sC2kdPh9NvoKFrIS40Tufe1ckq21FKYncxXr5nANacVHN9jH0Rk0FEiiFbVO2D93w5X/k3BuWZzJsLM6wOdu0ULaU/O4W/v7uFHf9/E+vKNjMtN5bsfmsXVs0cTH6sEIBIJlAiijTusfBj++n8Dd+xmjoUpV8K4c6FoIaSPAgJPAP3Lqj386KUlbK44wIQRw/jBDbO5YsYo4pQARCKKEkE0aa6Dv34WVv0+MI7/fd+HnFO6FGlt7+DPb+/ivsWb2FbVwOS8NH784TlcPn0UsZE6mbtIlFMiiBZ7VsHjt8G+LXD+l2HhZyHmcOduS1sHT7xVyn2LN1G6v5Fpo9P52c2nc8nUkeGZ4lFEBowSQaRzh+W/hL9/CVKy4V+eCTzWIaiptZ0/Lt/JT1/ezO6aJmYVZvKNq6ZxweQRJzfpi4gMGUoEkayxGp75BKx5CiZcDB/4WeA5PkBjSzuPvrmD+1/ZTHltM6ePzeL/XTuTcybmKgGIRJmQJgIzuwz4ARAL/MLd7+m2/V7g/OBiCjDC3TNDGVPU2LUC/ngb1O6Ci++Gsz5+6Lk+NQ2tfPBnr7Nxbz3zx2dz73WzOeuUHCUAkSgVskRgZrHAfcDFQCmwzMyedvc1B8u4+6c7lf84MCdU8UQNd3jjJ/D81yAtD277GxTOO7S5pa2Dj/5uBduqDvDgrXO5YPLIMAYrIoNBKK8I5gGb3H0LgJk9BlwNrDlK+RuBr4UwnsjXsA/+/DHY8HeY/D64+seBiVqC3J0vPbmapVuquPf6WUoCIgKENhHkAzs7LZcCZ/ZU0MzGAuOAl0IYT2TbvhSe+EhgQvbLvwPzFh3xTP/7Fm/i8RWlfPLCiXxgTkGYAhWRwWawdBbfADzu7u09bTSzRcAigDFjxgxkXINfRwf841546VuQOQY+8hyMPrKF7amVu/jucxv4wJx8PnXRxDAEKiKDVSgTwS6gsNNyQXBdT24A7jjagdz9AeABgLlz53p/BTjk1e+FPy2CLYth+rWBG8SS0o8otnzbPj73x1XMK8rmnmtnqFNYRLoIZSJYBkw0s3EEEsANwIe7FzKzyUAWsDSEsUSeLS8HkkBTDVz5Qzjt//Q4veO2ygPc/pvl5Gclc/8tp5MYpyeEikhXIUsE7t5mZncCxQSGjz7o7iVmdjew3N2fDha9AXjM3XWm3xftbbDkv+GV/4HcU+GWJ2HktB6LVje08K8PLQPgV7eeQVaqJokRkSOFtI/A3Z8Fnu227qvdlr8eyhgiSu1ueOLfYPs/YPbN8N7vQEJqj0Wb29pZ9NsVlO5v5OHbz6Qot+dyIiKDpbNYerPhOfjzR6G1CT5wP8y64ahF3Z0vPrGaN7fu4wc3zOaMouwBDFREhholgsGuvRVevBte/2FgkpgPPQS5xx7188MXN/Gnt3fxmYtP5erZ+QMTp4gMWUoEg9n+7YF7A0qXwdyPwKXfgvjkY+7y57d3ce8LG7jmtHw+fsGEAQpURIYyJYLBau0z8NQdgUdGfOghmPaBXnd5c+s+Pv/4KuaPz+aea2ZqmKiI9IkSwWDT1gzP/Se8eT+MPg0++CBkj+t1t62VB1j02+UUZCXzs5tP1zzCItJnSgSDSdXmwOQxe96Bs+6EC78Gcb0P+dx/oIXbfvUmMWb86rYzyEzRMFER6TslgsFi9ePwzKcgNg5ufAwmXd6n3QLDRJezu6aJR/7tTMbmaJioiBwfJYJwa2mAv98Fb/0aCufDB38JGX17IJy78/nHV7Fs235+eOMc5mqYqIicACWCcNq7LtAUtHdtYA7h874UuCLoo++/sJGnVu7mc5dO4qpZo0MYqIhEMiWCcHCHlQ/Ds58L3Bl88xMw4cLjOsQTK0r5wYsb+dDpBfzHeaeEKFARiQZKBAOtuR7++hlY9XsYdw5c8/PATGLH4Y0tVdz1p1WcNT6Hb31ATxMVkZOjRDDQnvx3WP8snP/lQHNQzPE9DXRzRT3//tsVjMlO0TBREekXSgQDadcKWPeXQBI49/PHvXtVfTO3/WoZcTHGr26dR0ZKfAiCFJFoo0QwkF76JqTkwPyPHfeuTa2Bp4mW1zbx6KL5jMlJCUGAIhKN1K4wULb9Aza/BGd/GhLTjmvXjg7nc4+vYsX2/XzvutmcNiar951ERPpIiWAguAeuBoblBR4ed5zufWEDz7yzm89fNokrZo4KQYAiEs2UCAbClsWw43U45/9CwvE16fxx+U5+9NImrp9byMfO1TBREel/SgSh5g4v/hdkFAbmFT4Or2+q5It/Ws2CCTl88wPTNUxUREJCiSDU1v8Ndr8VGCUUl9jn3Tbtreejv1vBuNxUfnLT6cTH6j+ViISGapdQ6uiAxd+C7PEw68N93q2qvpnbHnqThLgYHrz1DDKSNUxUREJHw0dDac2fofxduOYXfX6GUFNrO7f/Zjl7a5t5bNF8CrM1TFREQkuJIFTa22Dxt2H4FJh+TZ926ehwPvvHd3hrRzU/vek05miYqIgMACWCUFn9B6jaCNf9ts+Pkfjuc+v566o9fPHyyVw+Q8NERWRgqI8gFNpa4OV7YNQsmHJln3b5/bId/OTlzdw4bwyLzhkf4gBFRA7TFUEorPwdVG+HK/4X+jDk8x+bKvnyk++ycGIud189TcNERWRA6Yqgv7U2wZL/gcIzYcJFvRbfWF7HR3+3gvHDU7nvptM0TFREBlxIax0zu8zM1pvZJjO76yhlrjOzNWZWYmaPhDKeAbH8QajbDRd8pdergYq6Zm57aBmJcbE8eOsZpCdpmKiIDLyQNQ2ZWSxwH3AxUAosM7On3X1NpzITgS8CC9x9v5mNCFU8A6K5Hl77XmDCmXHnHLNoY0s7//ab5VTWN/P7RWdRkKVhoiISHqG8IpgHbHL3Le7eAjwGXN2tzO3Afe6+H8Dd94YwntB78wE4UAEX/Ocxi3V0OJ/5w0pWlVbz/evnMKswc2DiExHpQSgTQT6ws9NyaXBdZ6cCp5rZP8zsDTO7rKcDmdkiM1tuZssrKipCFO5JaqqBf/wAJl4KhfOOWfS/i9fxt3fL+PJ7p3DZ9OObplJEpL+Fu2cyDpgInAfcCPzczDK7F3L3B9x9rrvPHT58+MBG2FdLfwJN1XD+l45Z7E9vlXL/ki3cdOYYPnL2uIGJTUTkGEKZCHYBhZ2WC4LrOisFnnb3VnffCmwgkBiGloZ9sPQ+mHIVjJ59zKIPvLKF6fnpfOMqDRMVkcEhlIlgGTDRzMaZWQJwA/B0tzJ/JnA1gJnlEmgq2hLCmELjH9+Hlvperwa2Vx1gXVkd75+dT5yGiYrIIBGy2sjd24A7gWJgLfAHdy8xs7vN7KpgsWKgyszWAIuBz7l7VahiCom6cvjnAzDzOhgx5ZhFi0vKALh0mvoFRGTwCOmdxe7+LPBst3Vf7fTegc8E/4am174H7S1w7hd6LVpcUs6UUel6oqiIDCpqnzgZ1TsDN5DNuQlyjj2N5N66Jt7asZ9Lp40coOBERPpGieBkvPI/gddzPt9r0efXlOOuZiERGXxOKBGY2bD+DmTIqdoMb/8OTr8VMgt7LV5cUs6Y7BQm56WFPjYRkeNwolcEa3ovEuGWfAdiE2DhZ3stWtvUytLNlVw6baSGjIrIoHPUzmIzO1oHrgHRfUWwdx2s+j285+OQ1ntTz+J1e2ltdzULicigdKwrgm8DWUBat79hvewX+V7+NiSkwoJP9al4cUkZucMSOU1TT4rIIHSs4aNvAX929xXdN5jZv4UupEFuzzuw5qlAB3FqTq/Fm1rbeXl9BVfPzicmRs1CIjL4HOvMfhew3cw+2cO2uSGKZ/Bb/G1IyoSz7uhT8dc2VtLQ0q5hoyIyaB0rEUwFEoB/NbMsM8s++Ae0Dkx4g8zOZbDh77DgE5Cc2addikvKSEuM4z2n5IY2NhGRE3SspqH7gReB8cAKAp3EB3lwfXR56b8gJRfm/Xufire1d/DC2nLOnzyChLjo7lYRkcHrqLWTu//Q3acAD7r7eHcf1+kv+pLA1ldg6xJY+BlI7NugqWXb9rO/oVWjhURkUOv1NNXdPzYQgQxq7vDStyBtFMz91z7vVlxSRkJcDOdNGqRzKIiIEO3DQPtq04uw8w0453MQn9ynXdyd59eUs3BCLqmJIX22n4jISVEi6I17oG8gcwzMuaXPu727q5Zd1Y1qFhKRQU+JoDfr/gJ7VsK5d0FcQp93Ky4pI8bgwikjQhebiEg/UCI4lo72QN9AzgSYef1x7VpcUsYZRdnkDEsMUXAiIv1DieBYSp6EirVw3hchtu/t/Fsq6tm4t17NQiIyJCgRHE17W+Au4hHTYNo1x7VrcUk5AJfobmIRGQI0nOVo3nkU9m2GGx6BmOPLl8UlZUzPT6cgS1NSisjgpyuCnrQ1w5L/htFzYNJ7j2vXspomVu6s5tKpahYSkaFBVwQ9ees3ULMTrvw+HOdEMs+vKQPg0ulKBCIyNOiKoLvWRnjluzDmLDjlwuPevbiknHG5qUwcEd1z94jI0KFE0N2yX0B9GVzwn8d9NVDT0MobW6q4dFqepqQUkSFDiaCz5jp47V4Yfz4ULTju3V9cV05bh2vuAREZUpQIOvvnz6ChCi74ygntXlxSxsj0RGYVZPZvXCIiIaREcFDjfvjHj+DUy6Hg+Cdga2xpZ8mGCi6ZmqcpKUVkSAlpIjCzy8xsvZltMrO7eth+q5lVmNnK4F/45kJeeh8018AFXz6h3V/ZWEFTa4fuJhaRISdkw0fNLBa4D7gYKAWWmdnT7r6mW9Hfu/udoYqjTw5Uwhs/hWkfgLwZJ3SI4pIyMpLjOXN8dj8HJyISWqG8IpgHbHL3Le7eAjwGXB3Czztxr90LrQ2BZwqdgNb2Dl5cu5cLJ48gPlatbSIytISy1soHdnZaLg2u6+5aM1tlZo+bWWFPBzKzRWa23MyWV1RU9G+UtXsCQ0ZnXg/DJ53QId7cuo+axlYuUbOQiAxB4T59fQYocveZwPPAr3sq5O4PuPtcd587fHg/T/v46v9CRxuc+4UTPkRxSRlJ8TGce6qmpBSRoSeUiWAX0PkMvyC47hB3r3L35uDiL4DTQxjPkfZvhxUPBWYeyx53Qofo6HCeKynnnInDSU6I7d/4REQGQCgTwTJgopmNM7ME4Abg6c4FzGxUp8WrgLUhjOdIr3wHLCYwF/EJWrWrhrLaJo0WEpEhK2Sjhty9zczuBIqBWOBBdy8xs7uB5e7+NPAJM7sKaAP2AbeGKp4jVG6ClY/CvEWQ0VPXRd8Ul5QRG2OaklJEhqyQPn3U3Z8Fnu227qud3n8ROLGhOidryT0Qlwhnf/qkDlNcUsb88dlkpvR9PmMRkcEk3J3F4VG+BlY/HrgaSDvx5wJt2lvHlooDahYSkSEtOhPB4m9BYhos+ORJHebQlJSahEZEhrDoSwS734Z1f4Gz7oCUk7sLuLikjFmFmeRlJPVTcCIiAy/6EsFL34LkLJj/sZM6zO7qRlaV1uiR0yIy5EVXItjxBmx6PtAklJRxUod6riQ4JaX6B0RkiIueROAOL30TUkcEOolPUnFJORNGDOOU4ZqSUkSGtuhJBFuXwLZXYeFnISH1pA61/0ALb27bp2YhEYkI0ZMIanbB8Mlw+q0nfagX1pbT3uFqFhKRiBDSG8oGlTk3wawbIebkc19xSTmjM5KYkX9y/QwiIoNB9FwRQL8kgYaWNl7dWMEl0/Iw05SUIjL0RVci6AdL1lfQ3NbBJeofEJEIoURwnIpLyshKiWdekaakFJHIoERwHFraOnhx3V4unDKSOE1JKSIRQrXZcVi6pYq6pjaNFhKRiKJEcByKS8pISYhl4cTccIciItJvlAj6qKPDeX5NOeeeOpykeE1JKSKRQ4mgj97euZ+KumY1C4lIxFEi6KPiknLiYozzJ2tKShGJLEoEfeDuFJeUcdYpOWQkx4c7HBGRfqVE0Afry+vYXtWgZiERiUhKBH1Q/G45ZnDJVN1NLCKRR4mgD4pLyjhtTBYj0jUlpYhEHiWCXuzc18CaPbWae0BEIpYSQS+KNSWliEQ4JYJePFdSzuS8NMbmnNysZiIig5USwTFU1jezbPs+LtHVgIhEsJAmAjO7zMzWm9kmM7vrGOWuNTM3s7mhjOd4vbCmHHfUPyAiES1kicDMYoH7gMuBqcCNZja1h3JpwCeBf4YqlhNVXFJGQVYyU0elhzsUEZGQCeUVwTxgk7tvcfcW4DHg6h7K/Rfw30BTCGM5bnVNrfxjUxWXakpKEYlwoUwE+cDOTsulwXWHmNlpQKG7//VYBzKzRWa23MyWV1RU9H+kPXh5fQUt7R0aLSQiES9sncVmFgN8D/hsb2Xd/QF3n+vuc4cPHx764Ag0C+WkJnD62KwB+TwRkXAJZSLYBRR2Wi4IrjsoDZgOvGxm24D5wNODocO4ua2dl9dXcPHUkcTGqFlIRCJbKBPBMmCimY0zswTgBuDpgxvdvcbdc929yN2LgDeAq9x9eQhj6pPXN1VR36wpKUUkOoQsEbh7G3AnUAysBf7g7iVmdreZXRWqz+0PxSVlDEuM4z0TcsIdiohIyMWF8uDu/izwbLd1Xz1K2fNCGUtftQenpDxv0nAS4zQlpYhEPt1Z3M2K7fupOtCiZiERiRpKBN0Ul5SREBvDeZMGZnSSiEi4KRF0cnBKygUTckhL0pSUIhIdlAg6WbOnltL9jWoWEpGookTQSXFJOTEGF2lKShGJIkoEnTxXUsbcsdnkDksMdygiIgNGiSBoe9UB1pXVcYkeOS0iUUaJIEhTUopItFIiCCouKWfqqHQKs1PCHYqIyIBSIgD21jXx1o79uhoQkaikRAA8f3BKyunqHxCR6KNEQKBZaGxOCpNGpoU7FBGRARf1iaC2qZWlmys1JaWIRK2oTwSL1+2ltd25VMNGRSRKRX0iKC4pY3haInMKNSWliESnqE4ETa2Hp6SM0ZSUIhKlojoRvLaxkoaWdg0bFZGoFtWJoLikjLSkOM4arykpRSR6RW0iaGvv4IW15VwweQQJcVH7M4iIRG8iWLZtP/sbWtUsJCJRL2oTQXFJGYlxMZx7qqakFJHoFpWJwN15fk05CycOJzUxLtzhiIiEVVQmgnd31bKrulE3kYmIEKWJoLikjNgY46IpSgQiIlGbCOYVZZOVmhDuUEREwi7qEsGWino27q1Xs5CISFBIE4GZXWZm681sk5nd1cP2j5rZajNbaWavmdnUUMYDgUdOA1yiYaMiIkAIE4GZxQL3AZcDU4Ebe6joH3H3Ge4+G/gO8L1QxXNQcUkZMwsyGJ2ZHOqPEhEZEkJ5RTAP2OTuW9y9BXgMuLpzAXev7bSYCngI46GspomVO6t1E5mISCehHESfD+zstFwKnNm9kJndAXwGSAAu6OlAZrYIWAQwZsyYEw7ouTVlAOofEBHpJOydxe5+n7ufAnwB+MpRyjzg7nPdfe7w4Sd+J3BxSRnjh6cyYYSmpBQROSiUiWAXUNhpuSC47mgeA94fqmCqG1p4Y8s+NQuJiHQTykSwDJhoZuPMLAG4AXi6cwEzm9hp8QpgY6iCeXHtXto7XIlARKSbkPURuHubmd0JFAOxwIPuXmJmdwPL3f1p4E4zuwhoBfYD/xKqeNKT47l46khm5meE6iNERIYkcw/pQJ1+N3fuXF++fHm4wxARGVLMbIW7z+1pW9g7i0VEJLyUCEREopwSgYhIlFMiEBGJckoEIiJRTolARCTKKRGIiEQ5JQIRkSg35G4oM7MKYPsJ7p4LVPZjOEOdfo+u9Hscpt+iq0j4Pca6e49P7RxyieBkmNnyo91ZF430e3Sl3+Mw/RZdRfrvoaYhEZEop0QgIhLloi0RPBDuAAYZ/R5d6fc4TL9FVxH9e0RVH4GIiBwp2q4IRESkGyUCEZEoFzWJwMwuM7P1ZrbJzO4KdzzhYmaFZrbYzNaYWYmZfTLcMQ0GZhZrZm+b2V/CHUu4mVmmmT1uZuvMbK2ZnRXumMLFzD4d/Hfyrpk9amZJ4Y4pFKIiEZhZLHAfcDkwFbjRzKaGN6qwaQM+6+5TgfnAHVH8W3T2SWBtuIMYJH4A/N3dJwOziNLfxczygU8Ac919OoEpd28Ib1ShERWJAJgHbHL3Le7eAjwGXB3mmMLC3fe4+1vB93UE/pHnhzeq8DKzAuAK4BfhjiXczCwDOAf4JYC7t7h7dViDCq84INnM4oAUYHeY4wmJaEkE+cDOTsulRHnlB2BmRcAc4J9hDiXcvg98HugIcxyDwTigAvhVsKnsF2aWGu6gwsHddwHfBXYAe4Aad38uvFGFRrQkAunGzIYBTwCfcvfacMcTLmb2PmCvu68IdyyDRBxwGvBTd58DHACisk/NzLIItByMA0YDqWZ2c3ijCo1oSQS7gMJOywXBdVHJzOIJJIGH3f1P4Y4nzBYAV5nZNgJNhheY2e/CG1JYlQKl7n7wKvFxAokhGl0EbHX3CndvBf4EvCfMMYVEtCSCZcBEMxtnZgkEOnyeDnNMYWFmRqD9d627fy/c8YSbu3/R3QvcvYjA/xcvuXtEnvX1hbuXATvNbFJw1YXAmjCGFE47gPlmlhL8d3MhEdpxHhfuAAaCu7eZ2Z1AMYGe/wfdvSTMYYXLAuAWYLWZrQyu+5K7Pxu+kGSQ+TjwcPCkaQtwW5jjCQt3/6eZPQ68RWC03dtE6KMm9IgJEZEoFy1NQyIichRKBCIiUU6JQEQkyikRiIhEOSUCEZEop0QgMoDM7Dw94VQGGyUCEZEop0Qg0gMzu9nM3jSzlWZ2f3C+gnozuzf4fPoXzWx4sOxsM3vDzFaZ2ZPBZ9RgZhPM7AUze8fM3jKzU4KHH9bpef8PB+9aFQkbJQKRbsxsCnA9sMDdZwPtwE1AKrDc3acBS4CvBXf5DfAFd58JrO60/mHgPnefReAZNXuC6+cAnyIwN8Z4And7i4RNVDxiQuQ4XQicDiwLnqwnA3sJPKb698EyvwP+FHx+f6a7Lwmu/zXwRzNLA/Ld/UkAd28CCB7vTXcvDS6vBIqA10L+rUSOQolA5EgG/Nrdv9hlpdl/dit3os9nae70vh39O5QwU9OQyJFeBD5oZiMAzCzbzMYS+PfywWCZDwOvuXsNsN/MFgbX3wIsCc7+Vmpm7w8eI9HMUgbyS4j0lc5ERLpx9zVm9hXgOTOLAVqBOwhM0jIvuG0vgX4EgH8Bfhas6Ds/rfMW4H4zuzt4jA8N4NcQ6TM9fVSkj8ys3t2HhTsOkf6mpiERkSinKwIRkSinKwIRkSinRCAiEuWUCEREopwSgYhIlFMiEBGJcv8fkjp81Z9jbKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(f1s)\n",
    "plt.plot(f1s_eval)\n",
    "plt.title('model f1 micro')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим еще максимальную f меру"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7923473419540229"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(f1s_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь попробуем сверточные слои. В торче сверточные слои не очень простые - нужно подробно разобраться в параметрах и расчитывать изменения в размере последовательности вручную. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, output_dim):\n",
    "        \n",
    "        super().__init__() \n",
    "        # параметры лучше передавать снаружи\n",
    "        # но для простоты напишем тут\n",
    "        self.VOCAB_SIZE = len(word2id)\n",
    "        self.EMB_DIM = 100 # размерность эмбединга слов\n",
    "        self.KERNEL_SIZE = 10 # размер фильтров в CNN, влияет на выходную длину последовательности\n",
    "        self.MAX_LEN = 200 # длина последовательности до которой западили \n",
    "        self.IN_CHANNELS = self.EMB_DIM # параметр который нужно передать в CNN, соответствующий размеру эмбединга\n",
    "        self.OUT_CHANNELS = 32 # количество фильтров в CNN, будет соответствовать размерности векторов на выходе из CNN\n",
    "        self.STRIDE = 1 # параметр CNN влияющий на выходную длину последовательности\n",
    "        self.HIDDEN_FC = 64 # перед финальной классификацией поставим еще один полносвязный слой\n",
    "        \n",
    "        \n",
    "        self.embedding = nn.Embedding(self.VOCAB_SIZE, self.EMB_DIM)\n",
    "        \n",
    "        self.conv = nn.Conv1d(self.IN_CHANNELS, \n",
    "                              self.OUT_CHANNELS, \n",
    "                              kernel_size=self.KERNEL_SIZE, \n",
    "                              stride=self.STRIDE, ) \n",
    "        \n",
    "        # к выходу из CNN и к выходу промежуточного полносвязного слоя нужно применить активацию\n",
    "        self.relu = nn.ReLU() \n",
    "        \n",
    "        \n",
    "        # расчитываем по формуле размерность получившейся последовательности\n",
    "        # умножаем на OUT_CHANNELS чтобы получить размер вектора\n",
    "        # который получиться если конкатенировать все в 1 вектор\n",
    "        \n",
    "        \n",
    "        self.flattened_size = int((((self.MAX_LEN-(self.KERNEL_SIZE-1)-1)/self.STRIDE)+1)*self.OUT_CHANNELS)\n",
    "        \n",
    "        # используем размер вектора для определения полносвязного слоя\n",
    "        self.fc = nn.Linear(self.flattened_size, \n",
    "                            self.HIDDEN_FC)\n",
    "        \n",
    "        self.fc_1 = nn.Linear(self.HIDDEN_FC, output_dim)\n",
    "        self.act = nn.LogSoftmax(1) \n",
    "        \n",
    "        \n",
    "    def forward(self, text):\n",
    "        \n",
    "        # сверточный слой ожидает на вход последовательность такого вида\n",
    "        # (batch_size, emb_dim, max_len)\n",
    "        # а у нас получается \n",
    "        # (batch_size, max_len, emb_dim)\n",
    "        # поэтому используем permute чтобы поменять местами max_len и emb_dim\n",
    "        embedded = self.embedding(text).permute(0, 2, 1)\n",
    "        \n",
    "        convoluted = self.relu(self.conv(embedded)) # пропускаем через свертку и применяем релу\n",
    "        hidden = torch.flatten(convoluted, 1, -1) # конкатенируем\n",
    "        hidden = self.relu(self.fc(hidden))  # подаем в полносвязный и применяем активацию\n",
    "        \n",
    "        dense_outputs = self.fc_1(hidden) \n",
    "        outputs=self.act(dense_outputs) \n",
    "        \n",
    "        return outputs\n",
    "\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(label2id))\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss() \n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 2.6197411823272705; F1 measure: 0.1808203125\n",
      "Loss: 2.4586977422237397; F1 measure: 0.19806640625\n",
      "Loss: 2.373960111141205; F1 measure: 0.20907552083333333\n",
      "Loss: 2.293777464926243; F1 measure: 0.235419921875\n",
      "Loss: 2.1999969239234924; F1 measure: 0.2706328125\n",
      "Eval -  0.46314655172413793\n",
      "1\n",
      "Loss: 1.5232779133319854; F1 measure: 0.5215234375\n",
      "Loss: 1.4571399873495101; F1 measure: 0.54337890625\n",
      "Loss: 1.4014333546161652; F1 measure: 0.5609895833333334\n",
      "Loss: 1.355219143331051; F1 measure: 0.57517578125\n",
      "Loss: 1.313402940273285; F1 measure: 0.588875\n",
      "Eval -  0.6512631704980844\n",
      "2\n",
      "Loss: 1.0344296985864638; F1 measure: 0.6799609375\n",
      "Loss: 1.0047901949286462; F1 measure: 0.68927734375\n",
      "Loss: 0.9862580219904582; F1 measure: 0.6943098958333334\n",
      "Loss: 0.9652436164021492; F1 measure: 0.701083984375\n",
      "Loss: 0.9504220184087754; F1 measure: 0.705734375\n",
      "Eval -  0.7233494971264368\n",
      "3\n",
      "Loss: 0.8189972084760666; F1 measure: 0.74671875\n",
      "Loss: 0.812115375995636; F1 measure: 0.7473046875\n",
      "Loss: 0.8023925479253133; F1 measure: 0.7495442708333333\n",
      "Loss: 0.7957931512594223; F1 measure: 0.752353515625\n",
      "Loss: 0.7878573575019836; F1 measure: 0.7547265625\n",
      "Eval -  0.7482405411877395\n",
      "4\n",
      "Loss: 0.7163863027095795; F1 measure: 0.7746484375\n",
      "Loss: 0.7063124039769173; F1 measure: 0.77802734375\n",
      "Loss: 0.70201800207297; F1 measure: 0.7785416666666667\n",
      "Loss: 0.6974907685816288; F1 measure: 0.780703125\n",
      "Loss: 0.6953367521762848; F1 measure: 0.7814453125\n",
      "Eval -  0.7612601772030652\n",
      "5\n",
      "Loss: 0.6381947246193885; F1 measure: 0.801328125\n",
      "Loss: 0.6351038740575313; F1 measure: 0.8006640625\n",
      "Loss: 0.6329314880569776; F1 measure: 0.8010026041666667\n",
      "Loss: 0.6318752785772085; F1 measure: 0.80125\n",
      "Loss: 0.6284054705500602; F1 measure: 0.801921875\n",
      "Eval -  0.7726394875478927\n",
      "6\n",
      "Loss: 0.583585424721241; F1 measure: 0.8152734375\n",
      "Loss: 0.5796784365177154; F1 measure: 0.81775390625\n",
      "Loss: 0.5782570865750313; F1 measure: 0.8176432291666667\n",
      "Loss: 0.5759222892671824; F1 measure: 0.81875\n",
      "Loss: 0.5757328113317489; F1 measure: 0.8190625\n",
      "Eval -  0.7789146312260536\n",
      "7\n",
      "Loss: 0.5341010338068009; F1 measure: 0.831484375\n",
      "Loss: 0.5356463412940502; F1 measure: 0.8300390625\n",
      "Loss: 0.5386197704076767; F1 measure: 0.8299479166666667\n",
      "Loss: 0.5349517285823822; F1 measure: 0.83123046875\n",
      "Loss: 0.5324598643779754; F1 measure: 0.832328125\n",
      "Eval -  0.7886871408045977\n",
      "8\n",
      "Loss: 0.48952979385852813; F1 measure: 0.8453515625\n",
      "Loss: 0.4932489763200283; F1 measure: 0.8452734375\n",
      "Loss: 0.4943637558817863; F1 measure: 0.8448307291666667\n",
      "Loss: 0.49427898682653904; F1 measure: 0.8447265625\n",
      "Loss: 0.4922732102274895; F1 measure: 0.8455859375\n",
      "Eval -  0.7920222701149425\n",
      "9\n",
      "Loss: 0.46336703181266786; F1 measure: 0.85640625\n",
      "Loss: 0.46282783165574076; F1 measure: 0.85646484375\n",
      "Loss: 0.4612191939353943; F1 measure: 0.8561067708333333\n",
      "Loss: 0.4582536684721708; F1 measure: 0.8560546875\n",
      "Loss: 0.4556128804087639; F1 measure: 0.856703125\n",
      "Eval -  0.7923473419540229\n"
     ]
    }
   ],
   "source": [
    "f1s = []\n",
    "f1s_eval = []\n",
    "\n",
    "for i in range(10):\n",
    "    print(i)\n",
    "    f1s.append(train(model, training_generator, optimizer, criterion, 100))\n",
    "    evl = evaluate(model, valid_generator, criterion)\n",
    "    print('Eval - ', evl)\n",
    "    f1s_eval.append(evl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvG0lEQVR4nO3deXxV1bn/8c+TeSBzgEASCAgyT4qIRZzHWrXV1qHq7+rtldte7fxra4fbwdv25+3trZ1sq22tHRzaaq3a2sYJUStWQBEM80yAhCSQiczJ8/vjHCAJgQTIyUnO+b5fr7zO2Xuvvc9zjrKevddaey9zd0REJHrFhDsAEREJLyUCEZEop0QgIhLllAhERKKcEoGISJRTIhARiXJKBBKVzOwhM/tmH8tuM7OLjrIt2cyeMbMaM/tj/0Z5xGeNMbN6M4sN5edI9FEiEDk5HwRGAjnu/iEzG2VmT5vZbjNzMyvqrw9y9x3uPszd2/vrmCKgRCByssYCG9y9LbjcAfwduDZ8IXVlZnHhjkEGNyUCGbSCTTKfM7NVZnbAzH5pZiPN7G9mVmdmL5hZVqfyV5lZiZlVm9nLZjal07Y5ZvZWcL/fA0ndPut9ZrYyuO/rZjazD/F9A/gqcH2wyeYj7l7u7j8BlvX3dzSzouBVRlxwOdvMfhW8+thvZn8Orj/PzErN7AtmVgb8yswSzez7wbK7g+8T+xKjRD4lAhnsrgUuBk4FrgT+BnwJGE7g/99PAJjZqcCjwKeC254FnjGzBDNLAP4M/BbIBv5IpzN2M5sDPAj8O5AD3A883VtF6e5fA74N/D7YZPPLUH7HHvwWSAGmASOAezttyyPwXccCi4AvA/OB2cAsYB7wlROMVyKMEoEMdj8KnmXvAl4F/unub7t7E/AkMCdY7nrgr+7+vLu3At8FkoH3EKgA44Hvu3uruz9O1zP2RcD97v5Pd293918DzcH9BkJfv+MhZjYKuBz4qLvvD36vJZ2KdABfc/dmd28EbgLudve97l4BfAO4JdRfTIYGtR3KYFfe6X1jD8vDgu9HA9sPbnD3DjPbCeQD7cAu7/qExe2d3o8F/sXMPt5pXULwmAOhr9+xs0Jgn7vvP8oxK4KJ5KAuv0/w/UB9PxnkdEUgkWI3gQodADMzApXlLmAPkB9cd9CYTu93At9y98xOfynu/uhABH6CdgLZZpZ5lO3dHyvc5fch8P13hyAuGYKUCCRS/AG4wswuNLN44LMEmndeB5YCbcAnzCzezK4h0EZ+0M+Bj5rZmRaQamZXmFnaiQRiZknAwf6FxOByv3L3PQT6En5iZlnB73XOMXZ5FPiKmQ03s1wCndy/6++4ZGhSIpCI4O7rgZuBHwGVBDpdr3T3FndvAa4BbgX2EehP+FOnfZcDtwM/BvYDm4JlT1QjUB98vy64HAq3AK3Bz9hLoKP8aL4JLAdWAauBt4LrRDBNTCMiEt10RSAiEuWUCEREopwSgYhIlFMiEBGJckPuhrLc3FwvKioKdxgiIkPKihUrKt19eE/bhlwiKCoqYvny5eEOQ0RkSDGz7UfbpqYhEZEop0QgIhLllAhERKLckOsj6ElrayulpaU0NTX1XngIS0pKoqCggPj4+HCHIiIRJCISQWlpKWlpaRQVFdH1AZORw92pqqqitLSUcePGhTscEYkgEdE01NTURE5OTsQmAQAzIycnJ+KvekRk4EVEIgAiOgkcFA3fUUQGXkQ0DYmIRJq6plbKaprYU9N06PWCySOYUZDR75+lRNAPqqureeSRR/iP//iP49rvve99L4888giZmZmhCUxEBh13p7apjT01jV0q+bLg8sF19c1tXfYzg5xhCUoEg1V1dTU/+clPjkgEbW1txMUd/Sd+9tlnQx2aiAwgd6e6oTVQmdceruh3V3ddbmhp77KfGYxISyQvI5kJw4excGIuozKSyMtIDrymJzEyPYmEuNC05isR9IO77rqLzZs3M3v2bOLj40lKSiIrK4t169axYcMG3v/+97Nz506ampr45Cc/yaJFi4DDj8uor6/n8ssv5+yzz+b1118nPz+fp556iuTk5DB/MxE5yN3Zd6Cl01l7tzP62ib21DTS1NrRZb8Yg5HpSeRlJDE5L43zJ40IVvJJjMpIYlRGMsPTEomPDV+XbcQlgm88U8Ka3bX9esypo9P52pXTjrr9nnvu4d1332XlypW8/PLLXHHFFbz77ruHhnk++OCDZGdn09jYyBlnnMG1115LTk5Ol2Ns3LiRRx99lJ///Odcd911PPHEE9x88839+j1E5NhqGlrZvu8A26sa2LGvge1VB9ixryF4Rt9ES1vXSj42xsgLVvJTR6dz0ZQRh8/iM5IYnZFM7rAE4sJYyfdFxCWCwWDevHldxvr/8Ic/5MknnwRg586dbNy48YhEMG7cOGbPng3A6aefzrZt2wYqXJGo0dHhlNU2BSv6QIW/fV8DO/c1sL2qgZrG1i7lc4clMCY7hVmFmVze6Sw+LyOZ0RlJ5AxLJDZm6I/mi7hEcKwz94GSmpp66P3LL7/MCy+8wNKlS0lJSeG8887r8V6AxMTEQ+9jY2NpbAzVfOcika2ptZ3S/YGKvfOZ/fZ9DZTua6Sl/fBZfWyMUZCVzJjsFN43cxRjc1IYk50afE0hNTHiqsgeRce3DLG0tDTq6up63FZTU0NWVhYpKSmsW7eON954Y4CjE4k81Q0th87md1Qd6PS+gbLaridaqQmxjMlJ5dQRaVw8ZSSF2SmMzUlhbHYqozOTBn2zzUBQIugHOTk5LFiwgOnTp5OcnMzIkSMPbbvsssv42c9+xpQpU5g0aRLz588PY6QiQ0P7oSacA+zoVMlv3xdYrm3qOrRyRFoiY7JTeM+EHMYePKMPntXnpCboZsxemLuHO4bjMnfuXO8+Mc3atWuZMmVKmCIaWNH0XSWyNba0s/NQE06gU3ZHsMIv3d+1CSc+1ijIClTsY4Jn9IHXVAqzk0lJ0Dltb8xshbvP7Wmbfj0RCYmDwy0Pnc0H2+sPdtLurWvuUj4tMY4xOSlMHpXGJdPyDlX2Y7JTGJ2ZHBGdsoOVEoGInLC29g52Vzd1GXK5o9NInO53x+alJzEmJ4VzTx3O2JyUYHt9KmOzU8hMiVcTTpgoEYjIMdU3t7Gj03DLg00426sa2FXdSHvH4eblhLgYCrOSGZuTypnjsg8144zNSaEgK4Wk+NgwfhM5GiUCEcHdKd3fyLqyOtaX1bJpb/2hJp2qAy1dymamxDM2O4WZBRlcOWsUY7NTD3XM5qUnEaMmnCFHiUAkytQ0trK+rI51ZbWsK6tj3Z5aNpTXd2nGGZ2RxNicVC6eOpIxwaGWY7IDI3EykjVDXqRRIhCJUC1tHWyprGd9WR1r9wTO9NeX1bG75vA4+/SkOCaPSuea0/KZlJfG5Lx0Th05jLQkVfbRRIkgDIYNG0Z9fX24w5AI4R4Yc79uT13gDD9Y4W+uqKe1PdB+Hx9rnDJ8GPPGZTMpL53JeWlMHpVGXnqSOmhFiUBkKKlvbjvUrBN4DTTtdL7BanRGEpPy0jh/8ohAhZ+Xzrjc1JA9wliGvpAmAjO7DPgBEAv8wt3v6bZ9DPBrIDNY5i53H3IP6b/rrrsoLCzkjjvuAODrX/86cXFxLF68mP3799Pa2so3v/lNrr766jBHKkNFW3sH26oOBCv6QIW/vryWnfsOP4NqWGIck/LSeN+s0UzJS2NSXjqTRqaRkaJmHTk+Ibuz2MxigQ3AxUApsAy40d3XdCrzAPC2u//UzKYCz7p70bGO2+udxX+7C8pW9+M3AfJmwOX3HHXz22+/zac+9SmWLFkCwNSpUykuLiYjI4P09HQqKyuZP38+GzduxMxOqmlIdxZHnoaWNtbsrmX1rhre3VXLurJaNu6tP/TI49gYY3xuarANP3CGPykvjYKsZDXrSJ+F687iecAmd98SDOIx4GpgTacyDqQH32cAu0MYT8jMmTOHvXv3snv3bioqKsjKyiIvL49Pf/rTvPLKK8TExLBr1y7Ky8vJy8sLd7gSRo0t7azZU8vq0mpW76pl9a5qNu2t5+BQ/OFpiUwZlc6CCblMGhloxz9l+DCNv5eQCmUiyAd2dlouBc7sVubrwHNm9nEgFbiopwOZ2SJgEcCYMWOO/anHOHMPpQ996EM8/vjjlJWVcf311/Pwww9TUVHBihUriI+Pp6ioqMfHT0vkamptZ+2ewJn+6tIaVu+qYePe+kM3YOUOS2RmQQaXTx/FjPwMZhRkMDI9KcxRSzQKd2fxjcBD7v6/ZnYW8Fszm+7uXaYBcvcHgAcg0DQUhjh7df3113P77bdTWVnJkiVL+MMf/sCIESOIj49n8eLFbN++PdwhSgg1tbazrqwuWOkHzvY3lNcdqvRzUgOTjl8ydSTT8zOYWZDJyPRENe0MRh3t0NoIbU2Bv9Ym6DjYGe/g3sPr0bYF18NR9utlW/djD58EGQX9/pVDmQh2AYWdlguC6zr7CHAZgLsvNbMkIBfYG8K4QmLatGnU1dWRn5/PqFGjuOmmm7jyyiuZMWMGc+fOZfLkyeEOUfpJc1s76w9V+jWsKq1hQ3kdbcFKPzs1gen5GVw4eQQzCjKYkZ/BqAwN0zwh7a2HK+O2xuDrwQq6sdNrc6ftweVjbe9ynG77dLT2Hle4XPE9OOMj/X7YUCaCZcBEMxtHIAHcAHy4W5kdwIXAQ2Y2BUgCKkIYU0itXn24kzo3N5elS5f2WE73EAwdLW0dbCivY1WwaWf1rmrWl9UdGp+fmRLPjPwMFk0az8yCDKbnZ5CfqU7cXjXXQ3051O2BurJOr2Vdl1sPnPhnxCZAXDLEJUJ8UuD9oddkSMmGuKTA+7jEbtuTAtsObo852EdjYNbzKxx926H/HyxY9Gjbejl21uEpcPtTyBKBu7eZ2Z1AMYGhoQ+6e4mZ3Q0sd/engc8CPzezTxO4RrrVh9oECRIxWtsDlf7qQ5V+Dev21B16Ln56UhwzCzL5yNmBSn9GfoZG7nTX0hCoxI+o5Mu7VvYtPczoF5cEaaMgLQ9GzYRTL4XkrE4VcrfKPC4xWIknHX49VC6pU+UtvQlpH0HwnoBnu637aqf3a4AFoYxB5Gia29pZsX0/r22s5PXNVazZU3toyGZaUhwz8jO47ewiZuRnMDM/k8LsKK70WxsPV+L1ZUc5iy+D5poj941NDFTuaaNg5DSYcOHh5bQ8GJYXeE3K6HR2LAMp3J3F/cbdI/4fqS6WTo67s6G8nlc3VvDqxkre3LqPxtZ24mKM2YWZ3PqeokBHbn4GY7JTIvcpmq2N0FgNTTXQFHztvtxQ1fVMvqn6yOPEJhyu0IdPgvHnBZfzulb0SZmq4Ae5iEgESUlJVFVVkZOTE7HJwN2pqqoiKUnDC4/H3tomXttUyWsbK3ltU+WhWbHGD0/lurkFLJw4nDPHZw+th6x1tENzbe+V+dGW25uPcXAgYVigSSYtD3InwLiFMGzk4Yr94Gtylir4CBERiaCgoIDS0lIqKoZsP3OfJCUlUVDQ/0PHIkljSzv/3Fp1qOJfVxZoi85OTWDBhFwWTshlwcRc8jOTwxxpkDs07IOanVBTGjj7bqwOVt7VXSvzxprA++ZaDg077InFQnJmoKklKfiaUdB1ufP25Mzg+kxISofYIZQUpV9ERCKIj49n3LjQ9KbL4NbR4ZTsruWVjRW8trGSFdv309LeQUJcDGcUZfGFyyazcGIuU0elh6epp7UJancFKvlDfzu7Lrc1HrlffGrXyjq9AEZO76UyD75PSNWZuhyXiEgEEl1K9zfw2sZKXt1UyeubKtnfEBj3PWVUOrcuKOLsCbmcUZRNckKIR424w4HKIyv2zssHerglZlhe4Ax95LTAyJiMwsByRgGkjw40ueisXAaQEoEMenVNrSzdXMVrmyp5dWMlWysDY8tHpidyweSRLJyYy4IJuQxPS+zfD25thJpdx67ou7e3x6ccrtjzZnSt5A9W9HH9HKfISVIikEGnrb2Dd0qreWVDoJ1/5c5q2juclIRY5o/P4Zb5Y1k4MZcJI4ad3OCA5nrYv+3wX/dKvqGy2w4W6CjNKIBRs2DyFUdW9OpAlSFIiUDCzt3ZWnng0Bn/G5urqGtuI8ZgRkEmHzv3FM6emMtpY7KOb3KVjo5A5+uhyn5r14r/QLfBBQnDDlfso+cEK/dOFX3aKIhL6LfvLTJYKBFI2GyvOsAvXt3KS+v2sqs60GFamJ3M+2aN5pyJubznlNzeJ1lpaehauXep7Ld3bbqxmECFnlUEk94beM0qguxxkDlWZ/MStZQIZMBtrqjnvsWbeGrlbuJijPMnjeBj553Cwom5jM1J7VrYPfC4gn1be67w68u7lk9Ig+yiwA1Op17WtbLPKFQnrEgPlAhkwGwor+PHL23imVW7SYqL5V8XFHH7OeMZkeRQvQMqX4WN27pV9tu7Da+0w2f1Ey8OPIQrqyjwmj1OZ/UiJ0CJQEJuze5afrx4I8+uLiM1IZbPzR/GLXnbSdv9FPzytUAS6Cw+NVCp50yACRcdruiziiCzUKNuRPqZEoGEzOrSGn740kbeXrOB8xPX8fTY7UxrfofYt7cGCiRnQ9HZMOeWTmf2RZCaq7N6kQGkRCD97p0NW3j1+SdJ27OUu+LWckpSaWBDdTqMXQBnLoJx58CIqRBzHKOARCQklAjk5DXVwvbX2fPOczRvfJkZLVuYZU5rQhI29j1wyu2BB5flzYJY/S8nMtjoX6Ucv5YG2PkGbH0F3/oq7H4b83ayPZ5VMZOoHP9Rpp99JUljz9C4e5EhQIlAetfWDKXLYOsrsPXVwPuOVjosjg1xp/Jc65WsS5rNmQsv47qzTg39M35EpF8pEciR2lth99vBiv8V2PnPwCTfFoOPmsX2U2/loT2F/KG8gMyMTD52xSl8b24hSfFKACJDkRKBBCY6KVsdqPS3vQrbX4eW+sC2kdPh9NvoKFrIS40Tufe1ckq21FKYncxXr5nANacVHN9jH0Rk0FEiiFbVO2D93w5X/k3BuWZzJsLM6wOdu0ULaU/O4W/v7uFHf9/E+vKNjMtN5bsfmsXVs0cTH6sEIBIJlAiijTusfBj++n8Dd+xmjoUpV8K4c6FoIaSPAgJPAP3Lqj386KUlbK44wIQRw/jBDbO5YsYo4pQARCKKEkE0aa6Dv34WVv0+MI7/fd+HnFO6FGlt7+DPb+/ivsWb2FbVwOS8NH784TlcPn0UsZE6mbtIlFMiiBZ7VsHjt8G+LXD+l2HhZyHmcOduS1sHT7xVyn2LN1G6v5Fpo9P52c2nc8nUkeGZ4lFEBowSQaRzh+W/hL9/CVKy4V+eCTzWIaiptZ0/Lt/JT1/ezO6aJmYVZvKNq6ZxweQRJzfpi4gMGUoEkayxGp75BKx5CiZcDB/4WeA5PkBjSzuPvrmD+1/ZTHltM6ePzeL/XTuTcybmKgGIRJmQJgIzuwz4ARAL/MLd7+m2/V7g/OBiCjDC3TNDGVPU2LUC/ngb1O6Ci++Gsz5+6Lk+NQ2tfPBnr7Nxbz3zx2dz73WzOeuUHCUAkSgVskRgZrHAfcDFQCmwzMyedvc1B8u4+6c7lf84MCdU8UQNd3jjJ/D81yAtD277GxTOO7S5pa2Dj/5uBduqDvDgrXO5YPLIMAYrIoNBKK8I5gGb3H0LgJk9BlwNrDlK+RuBr4UwnsjXsA/+/DHY8HeY/D64+seBiVqC3J0vPbmapVuquPf6WUoCIgKENhHkAzs7LZcCZ/ZU0MzGAuOAl0IYT2TbvhSe+EhgQvbLvwPzFh3xTP/7Fm/i8RWlfPLCiXxgTkGYAhWRwWawdBbfADzu7u09bTSzRcAigDFjxgxkXINfRwf841546VuQOQY+8hyMPrKF7amVu/jucxv4wJx8PnXRxDAEKiKDVSgTwS6gsNNyQXBdT24A7jjagdz9AeABgLlz53p/BTjk1e+FPy2CLYth+rWBG8SS0o8otnzbPj73x1XMK8rmnmtnqFNYRLoIZSJYBkw0s3EEEsANwIe7FzKzyUAWsDSEsUSeLS8HkkBTDVz5Qzjt//Q4veO2ygPc/pvl5Gclc/8tp5MYpyeEikhXIUsE7t5mZncCxQSGjz7o7iVmdjew3N2fDha9AXjM3XWm3xftbbDkv+GV/4HcU+GWJ2HktB6LVje08K8PLQPgV7eeQVaqJokRkSOFtI/A3Z8Fnu227qvdlr8eyhgiSu1ueOLfYPs/YPbN8N7vQEJqj0Wb29pZ9NsVlO5v5OHbz6Qot+dyIiKDpbNYerPhOfjzR6G1CT5wP8y64ahF3Z0vPrGaN7fu4wc3zOaMouwBDFREhholgsGuvRVevBte/2FgkpgPPQS5xx7188MXN/Gnt3fxmYtP5erZ+QMTp4gMWUoEg9n+7YF7A0qXwdyPwKXfgvjkY+7y57d3ce8LG7jmtHw+fsGEAQpURIYyJYLBau0z8NQdgUdGfOghmPaBXnd5c+s+Pv/4KuaPz+aea2ZqmKiI9IkSwWDT1gzP/Se8eT+MPg0++CBkj+t1t62VB1j02+UUZCXzs5tP1zzCItJnSgSDSdXmwOQxe96Bs+6EC78Gcb0P+dx/oIXbfvUmMWb86rYzyEzRMFER6TslgsFi9ePwzKcgNg5ufAwmXd6n3QLDRJezu6aJR/7tTMbmaJioiBwfJYJwa2mAv98Fb/0aCufDB38JGX17IJy78/nHV7Fs235+eOMc5mqYqIicACWCcNq7LtAUtHdtYA7h874UuCLoo++/sJGnVu7mc5dO4qpZo0MYqIhEMiWCcHCHlQ/Ds58L3Bl88xMw4cLjOsQTK0r5wYsb+dDpBfzHeaeEKFARiQZKBAOtuR7++hlY9XsYdw5c8/PATGLH4Y0tVdz1p1WcNT6Hb31ATxMVkZOjRDDQnvx3WP8snP/lQHNQzPE9DXRzRT3//tsVjMlO0TBREekXSgQDadcKWPeXQBI49/PHvXtVfTO3/WoZcTHGr26dR0ZKfAiCFJFoo0QwkF76JqTkwPyPHfeuTa2Bp4mW1zbx6KL5jMlJCUGAIhKN1K4wULb9Aza/BGd/GhLTjmvXjg7nc4+vYsX2/XzvutmcNiar951ERPpIiWAguAeuBoblBR4ed5zufWEDz7yzm89fNokrZo4KQYAiEs2UCAbClsWw43U45/9CwvE16fxx+U5+9NImrp9byMfO1TBREel/SgSh5g4v/hdkFAbmFT4Or2+q5It/Ws2CCTl88wPTNUxUREJCiSDU1v8Ndr8VGCUUl9jn3Tbtreejv1vBuNxUfnLT6cTH6j+ViISGapdQ6uiAxd+C7PEw68N93q2qvpnbHnqThLgYHrz1DDKSNUxUREJHw0dDac2fofxduOYXfX6GUFNrO7f/Zjl7a5t5bNF8CrM1TFREQkuJIFTa22Dxt2H4FJh+TZ926ehwPvvHd3hrRzU/vek05miYqIgMACWCUFn9B6jaCNf9ts+Pkfjuc+v566o9fPHyyVw+Q8NERWRgqI8gFNpa4OV7YNQsmHJln3b5/bId/OTlzdw4bwyLzhkf4gBFRA7TFUEorPwdVG+HK/4X+jDk8x+bKvnyk++ycGIud189TcNERWRA6Yqgv7U2wZL/gcIzYcJFvRbfWF7HR3+3gvHDU7nvptM0TFREBlxIax0zu8zM1pvZJjO76yhlrjOzNWZWYmaPhDKeAbH8QajbDRd8pdergYq6Zm57aBmJcbE8eOsZpCdpmKiIDLyQNQ2ZWSxwH3AxUAosM7On3X1NpzITgS8CC9x9v5mNCFU8A6K5Hl77XmDCmXHnHLNoY0s7//ab5VTWN/P7RWdRkKVhoiISHqG8IpgHbHL3Le7eAjwGXN2tzO3Afe6+H8Dd94YwntB78wE4UAEX/Ocxi3V0OJ/5w0pWlVbz/evnMKswc2DiExHpQSgTQT6ws9NyaXBdZ6cCp5rZP8zsDTO7rKcDmdkiM1tuZssrKipCFO5JaqqBf/wAJl4KhfOOWfS/i9fxt3fL+PJ7p3DZ9OObplJEpL+Fu2cyDpgInAfcCPzczDK7F3L3B9x9rrvPHT58+MBG2FdLfwJN1XD+l45Z7E9vlXL/ki3cdOYYPnL2uIGJTUTkGEKZCHYBhZ2WC4LrOisFnnb3VnffCmwgkBiGloZ9sPQ+mHIVjJ59zKIPvLKF6fnpfOMqDRMVkcEhlIlgGTDRzMaZWQJwA/B0tzJ/JnA1gJnlEmgq2hLCmELjH9+Hlvperwa2Vx1gXVkd75+dT5yGiYrIIBGy2sjd24A7gWJgLfAHdy8xs7vN7KpgsWKgyszWAIuBz7l7VahiCom6cvjnAzDzOhgx5ZhFi0vKALh0mvoFRGTwCOmdxe7+LPBst3Vf7fTegc8E/4am174H7S1w7hd6LVpcUs6UUel6oqiIDCpqnzgZ1TsDN5DNuQlyjj2N5N66Jt7asZ9Lp40coOBERPpGieBkvPI/gddzPt9r0efXlOOuZiERGXxOKBGY2bD+DmTIqdoMb/8OTr8VMgt7LV5cUs6Y7BQm56WFPjYRkeNwolcEa3ovEuGWfAdiE2DhZ3stWtvUytLNlVw6baSGjIrIoHPUzmIzO1oHrgHRfUWwdx2s+j285+OQ1ntTz+J1e2ltdzULicigdKwrgm8DWUBat79hvewX+V7+NiSkwoJP9al4cUkZucMSOU1TT4rIIHSs4aNvAX929xXdN5jZv4UupEFuzzuw5qlAB3FqTq/Fm1rbeXl9BVfPzicmRs1CIjL4HOvMfhew3cw+2cO2uSGKZ/Bb/G1IyoSz7uhT8dc2VtLQ0q5hoyIyaB0rEUwFEoB/NbMsM8s++Ae0Dkx4g8zOZbDh77DgE5Cc2addikvKSEuM4z2n5IY2NhGRE3SspqH7gReB8cAKAp3EB3lwfXR56b8gJRfm/Xufire1d/DC2nLOnzyChLjo7lYRkcHrqLWTu//Q3acAD7r7eHcf1+kv+pLA1ldg6xJY+BlI7NugqWXb9rO/oVWjhURkUOv1NNXdPzYQgQxq7vDStyBtFMz91z7vVlxSRkJcDOdNGqRzKIiIEO3DQPtq04uw8w0453MQn9ynXdyd59eUs3BCLqmJIX22n4jISVEi6I17oG8gcwzMuaXPu727q5Zd1Y1qFhKRQU+JoDfr/gJ7VsK5d0FcQp93Ky4pI8bgwikjQhebiEg/UCI4lo72QN9AzgSYef1x7VpcUsYZRdnkDEsMUXAiIv1DieBYSp6EirVw3hchtu/t/Fsq6tm4t17NQiIyJCgRHE17W+Au4hHTYNo1x7VrcUk5AJfobmIRGQI0nOVo3nkU9m2GGx6BmOPLl8UlZUzPT6cgS1NSisjgpyuCnrQ1w5L/htFzYNJ7j2vXspomVu6s5tKpahYSkaFBVwQ9ees3ULMTrvw+HOdEMs+vKQPg0ulKBCIyNOiKoLvWRnjluzDmLDjlwuPevbiknHG5qUwcEd1z94jI0KFE0N2yX0B9GVzwn8d9NVDT0MobW6q4dFqepqQUkSFDiaCz5jp47V4Yfz4ULTju3V9cV05bh2vuAREZUpQIOvvnz6ChCi74ygntXlxSxsj0RGYVZPZvXCIiIaREcFDjfvjHj+DUy6Hg+Cdga2xpZ8mGCi6ZmqcpKUVkSAlpIjCzy8xsvZltMrO7eth+q5lVmNnK4F/45kJeeh8018AFXz6h3V/ZWEFTa4fuJhaRISdkw0fNLBa4D7gYKAWWmdnT7r6mW9Hfu/udoYqjTw5Uwhs/hWkfgLwZJ3SI4pIyMpLjOXN8dj8HJyISWqG8IpgHbHL3Le7eAjwGXB3Czztxr90LrQ2BZwqdgNb2Dl5cu5cLJ48gPlatbSIytISy1soHdnZaLg2u6+5aM1tlZo+bWWFPBzKzRWa23MyWV1RU9G+UtXsCQ0ZnXg/DJ53QId7cuo+axlYuUbOQiAxB4T59fQYocveZwPPAr3sq5O4PuPtcd587fHg/T/v46v9CRxuc+4UTPkRxSRlJ8TGce6qmpBSRoSeUiWAX0PkMvyC47hB3r3L35uDiL4DTQxjPkfZvhxUPBWYeyx53Qofo6HCeKynnnInDSU6I7d/4REQGQCgTwTJgopmNM7ME4Abg6c4FzGxUp8WrgLUhjOdIr3wHLCYwF/EJWrWrhrLaJo0WEpEhK2Sjhty9zczuBIqBWOBBdy8xs7uB5e7+NPAJM7sKaAP2AbeGKp4jVG6ClY/CvEWQ0VPXRd8Ul5QRG2OaklJEhqyQPn3U3Z8Fnu227qud3n8ROLGhOidryT0Qlwhnf/qkDlNcUsb88dlkpvR9PmMRkcEk3J3F4VG+BlY/HrgaSDvx5wJt2lvHlooDahYSkSEtOhPB4m9BYhos+ORJHebQlJSahEZEhrDoSwS734Z1f4Gz7oCUk7sLuLikjFmFmeRlJPVTcCIiAy/6EsFL34LkLJj/sZM6zO7qRlaV1uiR0yIy5EVXItjxBmx6PtAklJRxUod6riQ4JaX6B0RkiIueROAOL30TUkcEOolPUnFJORNGDOOU4ZqSUkSGtuhJBFuXwLZXYeFnISH1pA61/0ALb27bp2YhEYkI0ZMIanbB8Mlw+q0nfagX1pbT3uFqFhKRiBDSG8oGlTk3wawbIebkc19xSTmjM5KYkX9y/QwiIoNB9FwRQL8kgYaWNl7dWMEl0/Iw05SUIjL0RVci6AdL1lfQ3NbBJeofEJEIoURwnIpLyshKiWdekaakFJHIoERwHFraOnhx3V4unDKSOE1JKSIRQrXZcVi6pYq6pjaNFhKRiKJEcByKS8pISYhl4cTccIciItJvlAj6qKPDeX5NOeeeOpykeE1JKSKRQ4mgj97euZ+KumY1C4lIxFEi6KPiknLiYozzJ2tKShGJLEoEfeDuFJeUcdYpOWQkx4c7HBGRfqVE0Afry+vYXtWgZiERiUhKBH1Q/G45ZnDJVN1NLCKRR4mgD4pLyjhtTBYj0jUlpYhEHiWCXuzc18CaPbWae0BEIpYSQS+KNSWliEQ4JYJePFdSzuS8NMbmnNysZiIig5USwTFU1jezbPs+LtHVgIhEsJAmAjO7zMzWm9kmM7vrGOWuNTM3s7mhjOd4vbCmHHfUPyAiES1kicDMYoH7gMuBqcCNZja1h3JpwCeBf4YqlhNVXFJGQVYyU0elhzsUEZGQCeUVwTxgk7tvcfcW4DHg6h7K/Rfw30BTCGM5bnVNrfxjUxWXakpKEYlwoUwE+cDOTsulwXWHmNlpQKG7//VYBzKzRWa23MyWV1RU9H+kPXh5fQUt7R0aLSQiES9sncVmFgN8D/hsb2Xd/QF3n+vuc4cPHx764Ag0C+WkJnD62KwB+TwRkXAJZSLYBRR2Wi4IrjsoDZgOvGxm24D5wNODocO4ua2dl9dXcPHUkcTGqFlIRCJbKBPBMmCimY0zswTgBuDpgxvdvcbdc929yN2LgDeAq9x9eQhj6pPXN1VR36wpKUUkOoQsEbh7G3AnUAysBf7g7iVmdreZXRWqz+0PxSVlDEuM4z0TcsIdiohIyMWF8uDu/izwbLd1Xz1K2fNCGUtftQenpDxv0nAS4zQlpYhEPt1Z3M2K7fupOtCiZiERiRpKBN0Ul5SREBvDeZMGZnSSiEi4KRF0cnBKygUTckhL0pSUIhIdlAg6WbOnltL9jWoWEpGookTQSXFJOTEGF2lKShGJIkoEnTxXUsbcsdnkDksMdygiIgNGiSBoe9UB1pXVcYkeOS0iUUaJIEhTUopItFIiCCouKWfqqHQKs1PCHYqIyIBSIgD21jXx1o79uhoQkaikRAA8f3BKyunqHxCR6KNEQKBZaGxOCpNGpoU7FBGRARf1iaC2qZWlmys1JaWIRK2oTwSL1+2ltd25VMNGRSRKRX0iKC4pY3haInMKNSWliESnqE4ETa2Hp6SM0ZSUIhKlojoRvLaxkoaWdg0bFZGoFtWJoLikjLSkOM4arykpRSR6RW0iaGvv4IW15VwweQQJcVH7M4iIRG8iWLZtP/sbWtUsJCJRL2oTQXFJGYlxMZx7qqakFJHoFpWJwN15fk05CycOJzUxLtzhiIiEVVQmgnd31bKrulE3kYmIEKWJoLikjNgY46IpSgQiIlGbCOYVZZOVmhDuUEREwi7qEsGWino27q1Xs5CISFBIE4GZXWZm681sk5nd1cP2j5rZajNbaWavmdnUUMYDgUdOA1yiYaMiIkAIE4GZxQL3AZcDU4Ebe6joH3H3Ge4+G/gO8L1QxXNQcUkZMwsyGJ2ZHOqPEhEZEkJ5RTAP2OTuW9y9BXgMuLpzAXev7bSYCngI46GspomVO6t1E5mISCehHESfD+zstFwKnNm9kJndAXwGSAAu6OlAZrYIWAQwZsyYEw7ouTVlAOofEBHpJOydxe5+n7ufAnwB+MpRyjzg7nPdfe7w4Sd+J3BxSRnjh6cyYYSmpBQROSiUiWAXUNhpuSC47mgeA94fqmCqG1p4Y8s+NQuJiHQTykSwDJhoZuPMLAG4AXi6cwEzm9hp8QpgY6iCeXHtXto7XIlARKSbkPURuHubmd0JFAOxwIPuXmJmdwPL3f1p4E4zuwhoBfYD/xKqeNKT47l46khm5meE6iNERIYkcw/pQJ1+N3fuXF++fHm4wxARGVLMbIW7z+1pW9g7i0VEJLyUCEREopwSgYhIlFMiEBGJckoEIiJRTolARCTKKRGIiEQ5JQIRkSg35G4oM7MKYPsJ7p4LVPZjOEOdfo+u9Hscpt+iq0j4Pca6e49P7RxyieBkmNnyo91ZF430e3Sl3+Mw/RZdRfrvoaYhEZEop0QgIhLloi0RPBDuAAYZ/R5d6fc4TL9FVxH9e0RVH4GIiBwp2q4IRESkGyUCEZEoFzWJwMwuM7P1ZrbJzO4KdzzhYmaFZrbYzNaYWYmZfTLcMQ0GZhZrZm+b2V/CHUu4mVmmmT1uZuvMbK2ZnRXumMLFzD4d/Hfyrpk9amZJ4Y4pFKIiEZhZLHAfcDkwFbjRzKaGN6qwaQM+6+5TgfnAHVH8W3T2SWBtuIMYJH4A/N3dJwOziNLfxczygU8Ac919OoEpd28Ib1ShERWJAJgHbHL3Le7eAjwGXB3mmMLC3fe4+1vB93UE/pHnhzeq8DKzAuAK4BfhjiXczCwDOAf4JYC7t7h7dViDCq84INnM4oAUYHeY4wmJaEkE+cDOTsulRHnlB2BmRcAc4J9hDiXcvg98HugIcxyDwTigAvhVsKnsF2aWGu6gwsHddwHfBXYAe4Aad38uvFGFRrQkAunGzIYBTwCfcvfacMcTLmb2PmCvu68IdyyDRBxwGvBTd58DHACisk/NzLIItByMA0YDqWZ2c3ijCo1oSQS7gMJOywXBdVHJzOIJJIGH3f1P4Y4nzBYAV5nZNgJNhheY2e/CG1JYlQKl7n7wKvFxAokhGl0EbHX3CndvBf4EvCfMMYVEtCSCZcBEMxtnZgkEOnyeDnNMYWFmRqD9d627fy/c8YSbu3/R3QvcvYjA/xcvuXtEnvX1hbuXATvNbFJw1YXAmjCGFE47gPlmlhL8d3MhEdpxHhfuAAaCu7eZ2Z1AMYGe/wfdvSTMYYXLAuAWYLWZrQyu+5K7Pxu+kGSQ+TjwcPCkaQtwW5jjCQt3/6eZPQ68RWC03dtE6KMm9IgJEZEoFy1NQyIichRKBCIiUU6JQEQkyikRiIhEOSUCEZEop0QgMoDM7Dw94VQGGyUCEZEop0Qg0gMzu9nM3jSzlWZ2f3C+gnozuzf4fPoXzWx4sOxsM3vDzFaZ2ZPBZ9RgZhPM7AUze8fM3jKzU4KHH9bpef8PB+9aFQkbJQKRbsxsCnA9sMDdZwPtwE1AKrDc3acBS4CvBXf5DfAFd58JrO60/mHgPnefReAZNXuC6+cAnyIwN8Z4And7i4RNVDxiQuQ4XQicDiwLnqwnA3sJPKb698EyvwP+FHx+f6a7Lwmu/zXwRzNLA/Ld/UkAd28CCB7vTXcvDS6vBIqA10L+rUSOQolA5EgG/Nrdv9hlpdl/dit3os9nae70vh39O5QwU9OQyJFeBD5oZiMAzCzbzMYS+PfywWCZDwOvuXsNsN/MFgbX3wIsCc7+Vmpm7w8eI9HMUgbyS4j0lc5ERLpx9zVm9hXgOTOLAVqBOwhM0jIvuG0vgX4EgH8Bfhas6Ds/rfMW4H4zuzt4jA8N4NcQ6TM9fVSkj8ys3t2HhTsOkf6mpiERkSinKwIRkSinKwIRkSinRCAiEuWUCEREopwSgYhIlFMiEBGJcv8fkjp81Z9jbKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(f1s)\n",
    "plt.plot(f1s_eval)\n",
    "plt.title('model f1 micro')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max(f1s_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Пулинг"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вместе с CNN слоями часто используют пулинг слои. По сути это такие же сверточные слои, но без обучаемой части. В CNN фильтры обучаются под конкретную задачу, а в пулинге фильтры фиксированы (как правило это стандартная функция - максимум, среднее). Например, MaxPooling будет брать кусочек входной последовательности и передавать дальше максимальное значение в этом кусочке.\n",
    "\n",
    "Параметры пулинга практически такие же как и в CNN (kernel_size, stride), но нет in_channels, out_channels, так как размерность вектора никак не меняется. Расчитать длину последовательности после пулинга можно также как и в CNN - нужно знать длину входной последовательности, kernel_size и stride."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        self.VOCAB_SIZE = len(word2id)\n",
    "        self.EMB_DIM = 100\n",
    "        self.KERNEL_SIZE = 5\n",
    "        self.MAX_LEN = 200\n",
    "        self.IN_CHANNELS = self.EMB_DIM\n",
    "        self.OUT_CHANNELS = 128\n",
    "        self.STRIDE = 2\n",
    "        self.STRIDE_POOL = 2\n",
    "        self.KERNEL_SIZE_POOL = 2\n",
    "        self.HIDDEN_FC = 64\n",
    "        \n",
    "        \n",
    "        self.embedding = nn.Embedding(self.VOCAB_SIZE, self.EMB_DIM)\n",
    "        self.conv = nn.Conv1d(self.IN_CHANNELS, \n",
    "                              self.OUT_CHANNELS, \n",
    "                              kernel_size=self.KERNEL_SIZE, \n",
    "                              stride=self.STRIDE) \n",
    "        \n",
    "        # считаем длину после снн\n",
    "        self.conv_length = int((((self.MAX_LEN-(self.KERNEL_SIZE-1)-1)/self.STRIDE)+1))\n",
    "        \n",
    "        \n",
    "        # пулинг с максимумом\n",
    "        self.max_pool = nn.MaxPool1d(kernel_size=self.KERNEL_SIZE_POOL, \n",
    "                                     stride=self.STRIDE_POOL) \n",
    "        # также считаем длину после пулинга\n",
    "        self.pool_length = int((((self.conv_length-(self.KERNEL_SIZE_POOL-1)-1)/self.STRIDE_POOL)+1))\n",
    "        \n",
    "        \n",
    "        # функций активации много \n",
    "        # релу самая популярная\n",
    "        # у нее есть вариант с \"протечкой\"\n",
    "        # в нем отрицательные значений зануляются не полностью\n",
    "        # а с каким-то маленьким коэффициентом (0.05) например\n",
    "        self.relu = torch.nn.LeakyReLU()\n",
    "        \n",
    "        \n",
    "        # считаем размерность конкатенированного вектора\n",
    "        # умножаем размерность вектора после CNN на длину последовательности после пулинга\n",
    "        self.fc = nn.Linear(self.OUT_CHANNELS * self.pool_length, \n",
    "                            self.HIDDEN_FC)\n",
    "        self.fc_1 = nn.Linear(self.HIDDEN_FC, output_dim)\n",
    "        self.act = nn.LogSoftmax(1) \n",
    "        \n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text).permute(0, 2, 1)\n",
    "        convoluted = self.relu(self.conv(embedded))\n",
    "        pooled = self.max_pool(convoluted)\n",
    "        hidden = torch.flatten(pooled, 1, -1)\n",
    "        hidden = self.relu(self.fc(hidden))\n",
    "        dense_outputs=self.fc_1(hidden) \n",
    "        outputs=self.act(dense_outputs)\n",
    "        return outputs\n",
    "\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(label2id))\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss() \n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 2.6197411823272705; F1 measure: 0.1808203125\n",
      "Loss: 2.4586977422237397; F1 measure: 0.19806640625\n",
      "Loss: 2.373960111141205; F1 measure: 0.20907552083333333\n",
      "Loss: 2.293777464926243; F1 measure: 0.235419921875\n",
      "Loss: 2.1999969239234924; F1 measure: 0.2706328125\n",
      "Eval -  0.46314655172413793\n",
      "1\n",
      "Loss: 1.5232779133319854; F1 measure: 0.5215234375\n",
      "Loss: 1.4571399873495101; F1 measure: 0.54337890625\n",
      "Loss: 1.4014333546161652; F1 measure: 0.5609895833333334\n",
      "Loss: 1.355219143331051; F1 measure: 0.57517578125\n",
      "Loss: 1.313402940273285; F1 measure: 0.588875\n",
      "Eval -  0.6512631704980844\n",
      "2\n",
      "Loss: 1.0344296985864638; F1 measure: 0.6799609375\n",
      "Loss: 1.0047901949286462; F1 measure: 0.68927734375\n",
      "Loss: 0.9862580219904582; F1 measure: 0.6943098958333334\n",
      "Loss: 0.9652436164021492; F1 measure: 0.701083984375\n",
      "Loss: 0.9504220184087754; F1 measure: 0.705734375\n",
      "Eval -  0.7233494971264368\n",
      "3\n",
      "Loss: 0.8189972084760666; F1 measure: 0.74671875\n",
      "Loss: 0.812115375995636; F1 measure: 0.7473046875\n",
      "Loss: 0.8023925479253133; F1 measure: 0.7495442708333333\n",
      "Loss: 0.7957931512594223; F1 measure: 0.752353515625\n",
      "Loss: 0.7878573575019836; F1 measure: 0.7547265625\n",
      "Eval -  0.7482405411877395\n",
      "4\n",
      "Loss: 0.7163863027095795; F1 measure: 0.7746484375\n",
      "Loss: 0.7063124039769173; F1 measure: 0.77802734375\n",
      "Loss: 0.70201800207297; F1 measure: 0.7785416666666667\n",
      "Loss: 0.6974907685816288; F1 measure: 0.780703125\n",
      "Loss: 0.6953367521762848; F1 measure: 0.7814453125\n",
      "Eval -  0.7612601772030652\n",
      "5\n",
      "Loss: 0.6381947246193885; F1 measure: 0.801328125\n",
      "Loss: 0.6351038740575313; F1 measure: 0.8006640625\n",
      "Loss: 0.6329314880569776; F1 measure: 0.8010026041666667\n",
      "Loss: 0.6318752785772085; F1 measure: 0.80125\n",
      "Loss: 0.6284054705500602; F1 measure: 0.801921875\n",
      "Eval -  0.7726394875478927\n",
      "6\n",
      "Loss: 0.583585424721241; F1 measure: 0.8152734375\n",
      "Loss: 0.5796784365177154; F1 measure: 0.81775390625\n",
      "Loss: 0.5782570865750313; F1 measure: 0.8176432291666667\n",
      "Loss: 0.5759222892671824; F1 measure: 0.81875\n",
      "Loss: 0.5757328113317489; F1 measure: 0.8190625\n",
      "Eval -  0.7789146312260536\n",
      "7\n",
      "Loss: 0.5341010338068009; F1 measure: 0.831484375\n",
      "Loss: 0.5356463412940502; F1 measure: 0.8300390625\n",
      "Loss: 0.5386197704076767; F1 measure: 0.8299479166666667\n",
      "Loss: 0.5349517285823822; F1 measure: 0.83123046875\n",
      "Loss: 0.5324598643779754; F1 measure: 0.832328125\n",
      "Eval -  0.7886871408045977\n",
      "8\n",
      "Loss: 0.48952979385852813; F1 measure: 0.8453515625\n",
      "Loss: 0.4932489763200283; F1 measure: 0.8452734375\n",
      "Loss: 0.4943637558817863; F1 measure: 0.8448307291666667\n",
      "Loss: 0.49427898682653904; F1 measure: 0.8447265625\n",
      "Loss: 0.4922732102274895; F1 measure: 0.8455859375\n",
      "Eval -  0.7920222701149425\n",
      "9\n",
      "Loss: 0.46336703181266786; F1 measure: 0.85640625\n",
      "Loss: 0.46282783165574076; F1 measure: 0.85646484375\n",
      "Loss: 0.4612191939353943; F1 measure: 0.8561067708333333\n",
      "Loss: 0.4582536684721708; F1 measure: 0.8560546875\n",
      "Loss: 0.4556128804087639; F1 measure: 0.856703125\n",
      "Eval -  0.7923473419540229\n"
     ]
    }
   ],
   "source": [
    "f1s = []\n",
    "f1s_eval = []\n",
    "\n",
    "for i in range(10):\n",
    "    print(i)\n",
    "    f1s.append(train(model, training_generator, optimizer, criterion, 100))\n",
    "    evl = evaluate(model, valid_generator, criterion)\n",
    "    print('Eval - ', evl)\n",
    "    f1s_eval.append(evl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvG0lEQVR4nO3deXxV1bn/8c+TeSBzgEASCAgyT4qIRZzHWrXV1qHq7+rtldte7fxra4fbwdv25+3trZ1sq22tHRzaaq3a2sYJUStWQBEM80yAhCSQiczJ8/vjHCAJgQTIyUnO+b5fr7zO2Xuvvc9zjrKevddaey9zd0REJHrFhDsAEREJLyUCEZEop0QgIhLllAhERKKcEoGISJRTIhARiXJKBBKVzOwhM/tmH8tuM7OLjrIt2cyeMbMaM/tj/0Z5xGeNMbN6M4sN5edI9FEiEDk5HwRGAjnu/iEzG2VmT5vZbjNzMyvqrw9y9x3uPszd2/vrmCKgRCByssYCG9y9LbjcAfwduDZ8IXVlZnHhjkEGNyUCGbSCTTKfM7NVZnbAzH5pZiPN7G9mVmdmL5hZVqfyV5lZiZlVm9nLZjal07Y5ZvZWcL/fA0ndPut9ZrYyuO/rZjazD/F9A/gqcH2wyeYj7l7u7j8BlvX3dzSzouBVRlxwOdvMfhW8+thvZn8Orj/PzErN7AtmVgb8yswSzez7wbK7g+8T+xKjRD4lAhnsrgUuBk4FrgT+BnwJGE7g/99PAJjZqcCjwKeC254FnjGzBDNLAP4M/BbIBv5IpzN2M5sDPAj8O5AD3A883VtF6e5fA74N/D7YZPPLUH7HHvwWSAGmASOAezttyyPwXccCi4AvA/OB2cAsYB7wlROMVyKMEoEMdj8KnmXvAl4F/unub7t7E/AkMCdY7nrgr+7+vLu3At8FkoH3EKgA44Hvu3uruz9O1zP2RcD97v5Pd293918DzcH9BkJfv+MhZjYKuBz4qLvvD36vJZ2KdABfc/dmd28EbgLudve97l4BfAO4JdRfTIYGtR3KYFfe6X1jD8vDgu9HA9sPbnD3DjPbCeQD7cAu7/qExe2d3o8F/sXMPt5pXULwmAOhr9+xs0Jgn7vvP8oxK4KJ5KAuv0/w/UB9PxnkdEUgkWI3gQodADMzApXlLmAPkB9cd9CYTu93At9y98xOfynu/uhABH6CdgLZZpZ5lO3dHyvc5fch8P13hyAuGYKUCCRS/AG4wswuNLN44LMEmndeB5YCbcAnzCzezK4h0EZ+0M+Bj5rZmRaQamZXmFnaiQRiZknAwf6FxOByv3L3PQT6En5iZlnB73XOMXZ5FPiKmQ03s1wCndy/6++4ZGhSIpCI4O7rgZuBHwGVBDpdr3T3FndvAa4BbgX2EehP+FOnfZcDtwM/BvYDm4JlT1QjUB98vy64HAq3AK3Bz9hLoKP8aL4JLAdWAauBt4LrRDBNTCMiEt10RSAiEuWUCEREopwSgYhIlFMiEBGJckPuhrLc3FwvKioKdxgiIkPKihUrKt19eE/bhlwiKCoqYvny5eEOQ0RkSDGz7UfbpqYhEZEop0QgIhLllAhERKLckOsj6ElrayulpaU0NTX1XngIS0pKoqCggPj4+HCHIiIRJCISQWlpKWlpaRQVFdH1AZORw92pqqqitLSUcePGhTscEYkgEdE01NTURE5OTsQmAQAzIycnJ+KvekRk4EVEIgAiOgkcFA3fUUQGXkQ0DYmIRJq6plbKaprYU9N06PWCySOYUZDR75+lRNAPqqureeSRR/iP//iP49rvve99L4888giZmZmhCUxEBh13p7apjT01jV0q+bLg8sF19c1tXfYzg5xhCUoEg1V1dTU/+clPjkgEbW1txMUd/Sd+9tlnQx2aiAwgd6e6oTVQmdceruh3V3ddbmhp77KfGYxISyQvI5kJw4excGIuozKSyMtIDrymJzEyPYmEuNC05isR9IO77rqLzZs3M3v2bOLj40lKSiIrK4t169axYcMG3v/+97Nz506ampr45Cc/yaJFi4DDj8uor6/n8ssv5+yzz+b1118nPz+fp556iuTk5DB/MxE5yN3Zd6Cl01l7tzP62ib21DTS1NrRZb8Yg5HpSeRlJDE5L43zJ40IVvJJjMpIYlRGMsPTEomPDV+XbcQlgm88U8Ka3bX9esypo9P52pXTjrr9nnvu4d1332XlypW8/PLLXHHFFbz77ruHhnk++OCDZGdn09jYyBlnnMG1115LTk5Ol2Ns3LiRRx99lJ///Odcd911PPHEE9x88839+j1E5NhqGlrZvu8A26sa2LGvge1VB9ixryF4Rt9ES1vXSj42xsgLVvJTR6dz0ZQRh8/iM5IYnZFM7rAE4sJYyfdFxCWCwWDevHldxvr/8Ic/5MknnwRg586dbNy48YhEMG7cOGbPng3A6aefzrZt2wYqXJGo0dHhlNU2BSv6QIW/fV8DO/c1sL2qgZrG1i7lc4clMCY7hVmFmVze6Sw+LyOZ0RlJ5AxLJDZm6I/mi7hEcKwz94GSmpp66P3LL7/MCy+8wNKlS0lJSeG8887r8V6AxMTEQ+9jY2NpbAzVfOcika2ptZ3S/YGKvfOZ/fZ9DZTua6Sl/fBZfWyMUZCVzJjsFN43cxRjc1IYk50afE0hNTHiqsgeRce3DLG0tDTq6up63FZTU0NWVhYpKSmsW7eON954Y4CjE4k81Q0th87md1Qd6PS+gbLaridaqQmxjMlJ5dQRaVw8ZSSF2SmMzUlhbHYqozOTBn2zzUBQIugHOTk5LFiwgOnTp5OcnMzIkSMPbbvsssv42c9+xpQpU5g0aRLz588PY6QiQ0P7oSacA+zoVMlv3xdYrm3qOrRyRFoiY7JTeM+EHMYePKMPntXnpCboZsxemLuHO4bjMnfuXO8+Mc3atWuZMmVKmCIaWNH0XSWyNba0s/NQE06gU3ZHsMIv3d+1CSc+1ijIClTsY4Jn9IHXVAqzk0lJ0Dltb8xshbvP7Wmbfj0RCYmDwy0Pnc0H2+sPdtLurWvuUj4tMY4xOSlMHpXGJdPyDlX2Y7JTGJ2ZHBGdsoOVEoGInLC29g52Vzd1GXK5o9NInO53x+alJzEmJ4VzTx3O2JyUYHt9KmOzU8hMiVcTTpgoEYjIMdU3t7Gj03DLg00426sa2FXdSHvH4eblhLgYCrOSGZuTypnjsg8144zNSaEgK4Wk+NgwfhM5GiUCEcHdKd3fyLqyOtaX1bJpb/2hJp2qAy1dymamxDM2O4WZBRlcOWsUY7NTD3XM5qUnEaMmnCFHiUAkytQ0trK+rI51ZbWsK6tj3Z5aNpTXd2nGGZ2RxNicVC6eOpIxwaGWY7IDI3EykjVDXqRRIhCJUC1tHWyprGd9WR1r9wTO9NeX1bG75vA4+/SkOCaPSuea0/KZlJfG5Lx0Th05jLQkVfbRRIkgDIYNG0Z9fX24w5AI4R4Yc79uT13gDD9Y4W+uqKe1PdB+Hx9rnDJ8GPPGZTMpL53JeWlMHpVGXnqSOmhFiUBkKKlvbjvUrBN4DTTtdL7BanRGEpPy0jh/8ohAhZ+Xzrjc1JA9wliGvpAmAjO7DPgBEAv8wt3v6bZ9DPBrIDNY5i53H3IP6b/rrrsoLCzkjjvuAODrX/86cXFxLF68mP3799Pa2so3v/lNrr766jBHKkNFW3sH26oOBCv6QIW/vryWnfsOP4NqWGIck/LSeN+s0UzJS2NSXjqTRqaRkaJmHTk+Ibuz2MxigQ3AxUApsAy40d3XdCrzAPC2u//UzKYCz7p70bGO2+udxX+7C8pW9+M3AfJmwOX3HHXz22+/zac+9SmWLFkCwNSpUykuLiYjI4P09HQqKyuZP38+GzduxMxOqmlIdxZHnoaWNtbsrmX1rhre3VXLurJaNu6tP/TI49gYY3xuarANP3CGPykvjYKsZDXrSJ+F687iecAmd98SDOIx4GpgTacyDqQH32cAu0MYT8jMmTOHvXv3snv3bioqKsjKyiIvL49Pf/rTvPLKK8TExLBr1y7Ky8vJy8sLd7gSRo0t7azZU8vq0mpW76pl9a5qNu2t5+BQ/OFpiUwZlc6CCblMGhloxz9l+DCNv5eQCmUiyAd2dlouBc7sVubrwHNm9nEgFbiopwOZ2SJgEcCYMWOO/anHOHMPpQ996EM8/vjjlJWVcf311/Pwww9TUVHBihUriI+Pp6ioqMfHT0vkamptZ+2ewJn+6tIaVu+qYePe+kM3YOUOS2RmQQaXTx/FjPwMZhRkMDI9KcxRSzQKd2fxjcBD7v6/ZnYW8Fszm+7uXaYBcvcHgAcg0DQUhjh7df3113P77bdTWVnJkiVL+MMf/sCIESOIj49n8eLFbN++PdwhSgg1tbazrqwuWOkHzvY3lNcdqvRzUgOTjl8ydSTT8zOYWZDJyPRENe0MRh3t0NoIbU2Bv9Ym6DjYGe/g3sPr0bYF18NR9utlW/djD58EGQX9/pVDmQh2AYWdlguC6zr7CHAZgLsvNbMkIBfYG8K4QmLatGnU1dWRn5/PqFGjuOmmm7jyyiuZMWMGc+fOZfLkyeEOUfpJc1s76w9V+jWsKq1hQ3kdbcFKPzs1gen5GVw4eQQzCjKYkZ/BqAwN0zwh7a2HK+O2xuDrwQq6sdNrc6ftweVjbe9ynG77dLT2Hle4XPE9OOMj/X7YUCaCZcBEMxtHIAHcAHy4W5kdwIXAQ2Y2BUgCKkIYU0itXn24kzo3N5elS5f2WE73EAwdLW0dbCivY1WwaWf1rmrWl9UdGp+fmRLPjPwMFk0az8yCDKbnZ5CfqU7cXjXXQ3051O2BurJOr2Vdl1sPnPhnxCZAXDLEJUJ8UuD9oddkSMmGuKTA+7jEbtuTAtsObo852EdjYNbzKxx926H/HyxY9Gjbejl21uEpcPtTyBKBu7eZ2Z1AMYGhoQ+6e4mZ3Q0sd/engc8CPzezTxO4RrrVh9oECRIxWtsDlf7qQ5V+Dev21B16Ln56UhwzCzL5yNmBSn9GfoZG7nTX0hCoxI+o5Mu7VvYtPczoF5cEaaMgLQ9GzYRTL4XkrE4VcrfKPC4xWIknHX49VC6pU+UtvQlpH0HwnoBnu637aqf3a4AFoYxB5Gia29pZsX0/r22s5PXNVazZU3toyGZaUhwz8jO47ewiZuRnMDM/k8LsKK70WxsPV+L1ZUc5iy+D5poj941NDFTuaaNg5DSYcOHh5bQ8GJYXeE3K6HR2LAMp3J3F/cbdI/4fqS6WTo67s6G8nlc3VvDqxkre3LqPxtZ24mKM2YWZ3PqeokBHbn4GY7JTIvcpmq2N0FgNTTXQFHztvtxQ1fVMvqn6yOPEJhyu0IdPgvHnBZfzulb0SZmq4Ae5iEgESUlJVFVVkZOTE7HJwN2pqqoiKUnDC4/H3tomXttUyWsbK3ltU+WhWbHGD0/lurkFLJw4nDPHZw+th6x1tENzbe+V+dGW25uPcXAgYVigSSYtD3InwLiFMGzk4Yr94Gtylir4CBERiaCgoIDS0lIqKoZsP3OfJCUlUVDQ/0PHIkljSzv/3Fp1qOJfVxZoi85OTWDBhFwWTshlwcRc8jOTwxxpkDs07IOanVBTGjj7bqwOVt7VXSvzxprA++ZaDg077InFQnJmoKklKfiaUdB1ufP25Mzg+kxISofYIZQUpV9ERCKIj49n3LjQ9KbL4NbR4ZTsruWVjRW8trGSFdv309LeQUJcDGcUZfGFyyazcGIuU0elh6epp7UJancFKvlDfzu7Lrc1HrlffGrXyjq9AEZO76UyD75PSNWZuhyXiEgEEl1K9zfw2sZKXt1UyeubKtnfEBj3PWVUOrcuKOLsCbmcUZRNckKIR424w4HKIyv2zssHerglZlhe4Ax95LTAyJiMwsByRgGkjw40ueisXAaQEoEMenVNrSzdXMVrmyp5dWMlWysDY8tHpidyweSRLJyYy4IJuQxPS+zfD25thJpdx67ou7e3x6ccrtjzZnSt5A9W9HH9HKfISVIikEGnrb2Dd0qreWVDoJ1/5c5q2juclIRY5o/P4Zb5Y1k4MZcJI4ad3OCA5nrYv+3wX/dKvqGy2w4W6CjNKIBRs2DyFUdW9OpAlSFIiUDCzt3ZWnng0Bn/G5urqGtuI8ZgRkEmHzv3FM6emMtpY7KOb3KVjo5A5+uhyn5r14r/QLfBBQnDDlfso+cEK/dOFX3aKIhL6LfvLTJYKBFI2GyvOsAvXt3KS+v2sqs60GFamJ3M+2aN5pyJubznlNzeJ1lpaehauXep7Ld3bbqxmECFnlUEk94beM0qguxxkDlWZ/MStZQIZMBtrqjnvsWbeGrlbuJijPMnjeBj553Cwom5jM1J7VrYPfC4gn1be67w68u7lk9Ig+yiwA1Op17WtbLPKFQnrEgPlAhkwGwor+PHL23imVW7SYqL5V8XFHH7OeMZkeRQvQMqX4WN27pV9tu7Da+0w2f1Ey8OPIQrqyjwmj1OZ/UiJ0CJQEJuze5afrx4I8+uLiM1IZbPzR/GLXnbSdv9FPzytUAS6Cw+NVCp50yACRcdruiziiCzUKNuRPqZEoGEzOrSGn740kbeXrOB8xPX8fTY7UxrfofYt7cGCiRnQ9HZMOeWTmf2RZCaq7N6kQGkRCD97p0NW3j1+SdJ27OUu+LWckpSaWBDdTqMXQBnLoJx58CIqRBzHKOARCQklAjk5DXVwvbX2fPOczRvfJkZLVuYZU5rQhI29j1wyu2BB5flzYJY/S8nMtjoX6Ucv5YG2PkGbH0F3/oq7H4b83ayPZ5VMZOoHP9Rpp99JUljz9C4e5EhQIlAetfWDKXLYOsrsPXVwPuOVjosjg1xp/Jc65WsS5rNmQsv47qzTg39M35EpF8pEciR2lth99vBiv8V2PnPwCTfFoOPmsX2U2/loT2F/KG8gMyMTD52xSl8b24hSfFKACJDkRKBBCY6KVsdqPS3vQrbX4eW+sC2kdPh9NvoKFrIS40Tufe1ckq21FKYncxXr5nANacVHN9jH0Rk0FEiiFbVO2D93w5X/k3BuWZzJsLM6wOdu0ULaU/O4W/v7uFHf9/E+vKNjMtN5bsfmsXVs0cTH6sEIBIJlAiijTusfBj++n8Dd+xmjoUpV8K4c6FoIaSPAgJPAP3Lqj386KUlbK44wIQRw/jBDbO5YsYo4pQARCKKEkE0aa6Dv34WVv0+MI7/fd+HnFO6FGlt7+DPb+/ivsWb2FbVwOS8NH784TlcPn0UsZE6mbtIlFMiiBZ7VsHjt8G+LXD+l2HhZyHmcOduS1sHT7xVyn2LN1G6v5Fpo9P52c2nc8nUkeGZ4lFEBowSQaRzh+W/hL9/CVKy4V+eCTzWIaiptZ0/Lt/JT1/ezO6aJmYVZvKNq6ZxweQRJzfpi4gMGUoEkayxGp75BKx5CiZcDB/4WeA5PkBjSzuPvrmD+1/ZTHltM6ePzeL/XTuTcybmKgGIRJmQJgIzuwz4ARAL/MLd7+m2/V7g/OBiCjDC3TNDGVPU2LUC/ngb1O6Ci++Gsz5+6Lk+NQ2tfPBnr7Nxbz3zx2dz73WzOeuUHCUAkSgVskRgZrHAfcDFQCmwzMyedvc1B8u4+6c7lf84MCdU8UQNd3jjJ/D81yAtD277GxTOO7S5pa2Dj/5uBduqDvDgrXO5YPLIMAYrIoNBKK8I5gGb3H0LgJk9BlwNrDlK+RuBr4UwnsjXsA/+/DHY8HeY/D64+seBiVqC3J0vPbmapVuquPf6WUoCIgKENhHkAzs7LZcCZ/ZU0MzGAuOAl0IYT2TbvhSe+EhgQvbLvwPzFh3xTP/7Fm/i8RWlfPLCiXxgTkGYAhWRwWawdBbfADzu7u09bTSzRcAigDFjxgxkXINfRwf841546VuQOQY+8hyMPrKF7amVu/jucxv4wJx8PnXRxDAEKiKDVSgTwS6gsNNyQXBdT24A7jjagdz9AeABgLlz53p/BTjk1e+FPy2CLYth+rWBG8SS0o8otnzbPj73x1XMK8rmnmtnqFNYRLoIZSJYBkw0s3EEEsANwIe7FzKzyUAWsDSEsUSeLS8HkkBTDVz5Qzjt//Q4veO2ygPc/pvl5Gclc/8tp5MYpyeEikhXIUsE7t5mZncCxQSGjz7o7iVmdjew3N2fDha9AXjM3XWm3xftbbDkv+GV/4HcU+GWJ2HktB6LVje08K8PLQPgV7eeQVaqJokRkSOFtI/A3Z8Fnu227qvdlr8eyhgiSu1ueOLfYPs/YPbN8N7vQEJqj0Wb29pZ9NsVlO5v5OHbz6Qot+dyIiKDpbNYerPhOfjzR6G1CT5wP8y64ahF3Z0vPrGaN7fu4wc3zOaMouwBDFREhholgsGuvRVevBte/2FgkpgPPQS5xx7188MXN/Gnt3fxmYtP5erZ+QMTp4gMWUoEg9n+7YF7A0qXwdyPwKXfgvjkY+7y57d3ce8LG7jmtHw+fsGEAQpURIYyJYLBau0z8NQdgUdGfOghmPaBXnd5c+s+Pv/4KuaPz+aea2ZqmKiI9IkSwWDT1gzP/Se8eT+MPg0++CBkj+t1t62VB1j02+UUZCXzs5tP1zzCItJnSgSDSdXmwOQxe96Bs+6EC78Gcb0P+dx/oIXbfvUmMWb86rYzyEzRMFER6TslgsFi9ePwzKcgNg5ufAwmXd6n3QLDRJezu6aJR/7tTMbmaJioiBwfJYJwa2mAv98Fb/0aCufDB38JGX17IJy78/nHV7Fs235+eOMc5mqYqIicACWCcNq7LtAUtHdtYA7h874UuCLoo++/sJGnVu7mc5dO4qpZo0MYqIhEMiWCcHCHlQ/Ds58L3Bl88xMw4cLjOsQTK0r5wYsb+dDpBfzHeaeEKFARiQZKBAOtuR7++hlY9XsYdw5c8/PATGLH4Y0tVdz1p1WcNT6Hb31ATxMVkZOjRDDQnvx3WP8snP/lQHNQzPE9DXRzRT3//tsVjMlO0TBREekXSgQDadcKWPeXQBI49/PHvXtVfTO3/WoZcTHGr26dR0ZKfAiCFJFoo0QwkF76JqTkwPyPHfeuTa2Bp4mW1zbx6KL5jMlJCUGAIhKN1K4wULb9Aza/BGd/GhLTjmvXjg7nc4+vYsX2/XzvutmcNiar951ERPpIiWAguAeuBoblBR4ed5zufWEDz7yzm89fNokrZo4KQYAiEs2UCAbClsWw43U45/9CwvE16fxx+U5+9NImrp9byMfO1TBREel/SgSh5g4v/hdkFAbmFT4Or2+q5It/Ws2CCTl88wPTNUxUREJCiSDU1v8Ndr8VGCUUl9jn3Tbtreejv1vBuNxUfnLT6cTH6j+ViISGapdQ6uiAxd+C7PEw68N93q2qvpnbHnqThLgYHrz1DDKSNUxUREJHw0dDac2fofxduOYXfX6GUFNrO7f/Zjl7a5t5bNF8CrM1TFREQkuJIFTa22Dxt2H4FJh+TZ926ehwPvvHd3hrRzU/vek05miYqIgMACWCUFn9B6jaCNf9ts+Pkfjuc+v566o9fPHyyVw+Q8NERWRgqI8gFNpa4OV7YNQsmHJln3b5/bId/OTlzdw4bwyLzhkf4gBFRA7TFUEorPwdVG+HK/4X+jDk8x+bKvnyk++ycGIud189TcNERWRA6Yqgv7U2wZL/gcIzYcJFvRbfWF7HR3+3gvHDU7nvptM0TFREBlxIax0zu8zM1pvZJjO76yhlrjOzNWZWYmaPhDKeAbH8QajbDRd8pdergYq6Zm57aBmJcbE8eOsZpCdpmKiIDLyQNQ2ZWSxwH3AxUAosM7On3X1NpzITgS8CC9x9v5mNCFU8A6K5Hl77XmDCmXHnHLNoY0s7//ab5VTWN/P7RWdRkKVhoiISHqG8IpgHbHL3Le7eAjwGXN2tzO3Afe6+H8Dd94YwntB78wE4UAEX/Ocxi3V0OJ/5w0pWlVbz/evnMKswc2DiExHpQSgTQT6ws9NyaXBdZ6cCp5rZP8zsDTO7rKcDmdkiM1tuZssrKipCFO5JaqqBf/wAJl4KhfOOWfS/i9fxt3fL+PJ7p3DZ9OObplJEpL+Fu2cyDpgInAfcCPzczDK7F3L3B9x9rrvPHT58+MBG2FdLfwJN1XD+l45Z7E9vlXL/ki3cdOYYPnL2uIGJTUTkGEKZCHYBhZ2WC4LrOisFnnb3VnffCmwgkBiGloZ9sPQ+mHIVjJ59zKIPvLKF6fnpfOMqDRMVkcEhlIlgGTDRzMaZWQJwA/B0tzJ/JnA1gJnlEmgq2hLCmELjH9+Hlvperwa2Vx1gXVkd75+dT5yGiYrIIBGy2sjd24A7gWJgLfAHdy8xs7vN7KpgsWKgyszWAIuBz7l7VahiCom6cvjnAzDzOhgx5ZhFi0vKALh0mvoFRGTwCOmdxe7+LPBst3Vf7fTegc8E/4am174H7S1w7hd6LVpcUs6UUel6oqiIDCpqnzgZ1TsDN5DNuQlyjj2N5N66Jt7asZ9Lp40coOBERPpGieBkvPI/gddzPt9r0efXlOOuZiERGXxOKBGY2bD+DmTIqdoMb/8OTr8VMgt7LV5cUs6Y7BQm56WFPjYRkeNwolcEa3ovEuGWfAdiE2DhZ3stWtvUytLNlVw6baSGjIrIoHPUzmIzO1oHrgHRfUWwdx2s+j285+OQ1ntTz+J1e2ltdzULicigdKwrgm8DWUBat79hvewX+V7+NiSkwoJP9al4cUkZucMSOU1TT4rIIHSs4aNvAX929xXdN5jZv4UupEFuzzuw5qlAB3FqTq/Fm1rbeXl9BVfPzicmRs1CIjL4HOvMfhew3cw+2cO2uSGKZ/Bb/G1IyoSz7uhT8dc2VtLQ0q5hoyIyaB0rEUwFEoB/NbMsM8s++Ae0Dkx4g8zOZbDh77DgE5Cc2addikvKSEuM4z2n5IY2NhGRE3SspqH7gReB8cAKAp3EB3lwfXR56b8gJRfm/Xufire1d/DC2nLOnzyChLjo7lYRkcHrqLWTu//Q3acAD7r7eHcf1+kv+pLA1ldg6xJY+BlI7NugqWXb9rO/oVWjhURkUOv1NNXdPzYQgQxq7vDStyBtFMz91z7vVlxSRkJcDOdNGqRzKIiIEO3DQPtq04uw8w0453MQn9ynXdyd59eUs3BCLqmJIX22n4jISVEi6I17oG8gcwzMuaXPu727q5Zd1Y1qFhKRQU+JoDfr/gJ7VsK5d0FcQp93Ky4pI8bgwikjQhebiEg/UCI4lo72QN9AzgSYef1x7VpcUsYZRdnkDEsMUXAiIv1DieBYSp6EirVw3hchtu/t/Fsq6tm4t17NQiIyJCgRHE17W+Au4hHTYNo1x7VrcUk5AJfobmIRGQI0nOVo3nkU9m2GGx6BmOPLl8UlZUzPT6cgS1NSisjgpyuCnrQ1w5L/htFzYNJ7j2vXspomVu6s5tKpahYSkaFBVwQ9ees3ULMTrvw+HOdEMs+vKQPg0ulKBCIyNOiKoLvWRnjluzDmLDjlwuPevbiknHG5qUwcEd1z94jI0KFE0N2yX0B9GVzwn8d9NVDT0MobW6q4dFqepqQUkSFDiaCz5jp47V4Yfz4ULTju3V9cV05bh2vuAREZUpQIOvvnz6ChCi74ygntXlxSxsj0RGYVZPZvXCIiIaREcFDjfvjHj+DUy6Hg+Cdga2xpZ8mGCi6ZmqcpKUVkSAlpIjCzy8xsvZltMrO7eth+q5lVmNnK4F/45kJeeh8018AFXz6h3V/ZWEFTa4fuJhaRISdkw0fNLBa4D7gYKAWWmdnT7r6mW9Hfu/udoYqjTw5Uwhs/hWkfgLwZJ3SI4pIyMpLjOXN8dj8HJyISWqG8IpgHbHL3Le7eAjwGXB3Czztxr90LrQ2BZwqdgNb2Dl5cu5cLJ48gPlatbSIytISy1soHdnZaLg2u6+5aM1tlZo+bWWFPBzKzRWa23MyWV1RU9G+UtXsCQ0ZnXg/DJ53QId7cuo+axlYuUbOQiAxB4T59fQYocveZwPPAr3sq5O4PuPtcd587fHg/T/v46v9CRxuc+4UTPkRxSRlJ8TGce6qmpBSRoSeUiWAX0PkMvyC47hB3r3L35uDiL4DTQxjPkfZvhxUPBWYeyx53Qofo6HCeKynnnInDSU6I7d/4REQGQCgTwTJgopmNM7ME4Abg6c4FzGxUp8WrgLUhjOdIr3wHLCYwF/EJWrWrhrLaJo0WEpEhK2Sjhty9zczuBIqBWOBBdy8xs7uB5e7+NPAJM7sKaAP2AbeGKp4jVG6ClY/CvEWQ0VPXRd8Ul5QRG2OaklJEhqyQPn3U3Z8Fnu227qud3n8ROLGhOidryT0Qlwhnf/qkDlNcUsb88dlkpvR9PmMRkcEk3J3F4VG+BlY/HrgaSDvx5wJt2lvHlooDahYSkSEtOhPB4m9BYhos+ORJHebQlJSahEZEhrDoSwS734Z1f4Gz7oCUk7sLuLikjFmFmeRlJPVTcCIiAy/6EsFL34LkLJj/sZM6zO7qRlaV1uiR0yIy5EVXItjxBmx6PtAklJRxUod6riQ4JaX6B0RkiIueROAOL30TUkcEOolPUnFJORNGDOOU4ZqSUkSGtuhJBFuXwLZXYeFnISH1pA61/0ALb27bp2YhEYkI0ZMIanbB8Mlw+q0nfagX1pbT3uFqFhKRiBDSG8oGlTk3wawbIebkc19xSTmjM5KYkX9y/QwiIoNB9FwRQL8kgYaWNl7dWMEl0/Iw05SUIjL0RVci6AdL1lfQ3NbBJeofEJEIoURwnIpLyshKiWdekaakFJHIoERwHFraOnhx3V4unDKSOE1JKSIRQrXZcVi6pYq6pjaNFhKRiKJEcByKS8pISYhl4cTccIciItJvlAj6qKPDeX5NOeeeOpykeE1JKSKRQ4mgj97euZ+KumY1C4lIxFEi6KPiknLiYozzJ2tKShGJLEoEfeDuFJeUcdYpOWQkx4c7HBGRfqVE0Afry+vYXtWgZiERiUhKBH1Q/G45ZnDJVN1NLCKRR4mgD4pLyjhtTBYj0jUlpYhEHiWCXuzc18CaPbWae0BEIpYSQS+KNSWliEQ4JYJePFdSzuS8NMbmnNysZiIig5USwTFU1jezbPs+LtHVgIhEsJAmAjO7zMzWm9kmM7vrGOWuNTM3s7mhjOd4vbCmHHfUPyAiES1kicDMYoH7gMuBqcCNZja1h3JpwCeBf4YqlhNVXFJGQVYyU0elhzsUEZGQCeUVwTxgk7tvcfcW4DHg6h7K/Rfw30BTCGM5bnVNrfxjUxWXakpKEYlwoUwE+cDOTsulwXWHmNlpQKG7//VYBzKzRWa23MyWV1RU9H+kPXh5fQUt7R0aLSQiES9sncVmFgN8D/hsb2Xd/QF3n+vuc4cPHx764Ag0C+WkJnD62KwB+TwRkXAJZSLYBRR2Wi4IrjsoDZgOvGxm24D5wNODocO4ua2dl9dXcPHUkcTGqFlIRCJbKBPBMmCimY0zswTgBuDpgxvdvcbdc929yN2LgDeAq9x9eQhj6pPXN1VR36wpKUUkOoQsEbh7G3AnUAysBf7g7iVmdreZXRWqz+0PxSVlDEuM4z0TcsIdiohIyMWF8uDu/izwbLd1Xz1K2fNCGUtftQenpDxv0nAS4zQlpYhEPt1Z3M2K7fupOtCiZiERiRpKBN0Ul5SREBvDeZMGZnSSiEi4KRF0cnBKygUTckhL0pSUIhIdlAg6WbOnltL9jWoWEpGookTQSXFJOTEGF2lKShGJIkoEnTxXUsbcsdnkDksMdygiIgNGiSBoe9UB1pXVcYkeOS0iUUaJIEhTUopItFIiCCouKWfqqHQKs1PCHYqIyIBSIgD21jXx1o79uhoQkaikRAA8f3BKyunqHxCR6KNEQKBZaGxOCpNGpoU7FBGRARf1iaC2qZWlmys1JaWIRK2oTwSL1+2ltd25VMNGRSRKRX0iKC4pY3haInMKNSWliESnqE4ETa2Hp6SM0ZSUIhKlojoRvLaxkoaWdg0bFZGoFtWJoLikjLSkOM4arykpRSR6RW0iaGvv4IW15VwweQQJcVH7M4iIRG8iWLZtP/sbWtUsJCJRL2oTQXFJGYlxMZx7qqakFJHoFpWJwN15fk05CycOJzUxLtzhiIiEVVQmgnd31bKrulE3kYmIEKWJoLikjNgY46IpSgQiIlGbCOYVZZOVmhDuUEREwi7qEsGWino27q1Xs5CISFBIE4GZXWZm681sk5nd1cP2j5rZajNbaWavmdnUUMYDgUdOA1yiYaMiIkAIE4GZxQL3AZcDU4Ebe6joH3H3Ge4+G/gO8L1QxXNQcUkZMwsyGJ2ZHOqPEhEZEkJ5RTAP2OTuW9y9BXgMuLpzAXev7bSYCngI46GspomVO6t1E5mISCehHESfD+zstFwKnNm9kJndAXwGSAAu6OlAZrYIWAQwZsyYEw7ouTVlAOofEBHpJOydxe5+n7ufAnwB+MpRyjzg7nPdfe7w4Sd+J3BxSRnjh6cyYYSmpBQROSiUiWAXUNhpuSC47mgeA94fqmCqG1p4Y8s+NQuJiHQTykSwDJhoZuPMLAG4AXi6cwEzm9hp8QpgY6iCeXHtXto7XIlARKSbkPURuHubmd0JFAOxwIPuXmJmdwPL3f1p4E4zuwhoBfYD/xKqeNKT47l46khm5meE6iNERIYkcw/pQJ1+N3fuXF++fHm4wxARGVLMbIW7z+1pW9g7i0VEJLyUCEREopwSgYhIlFMiEBGJckoEIiJRTolARCTKKRGIiEQ5JQIRkSg35G4oM7MKYPsJ7p4LVPZjOEOdfo+u9Hscpt+iq0j4Pca6e49P7RxyieBkmNnyo91ZF430e3Sl3+Mw/RZdRfrvoaYhEZEop0QgIhLloi0RPBDuAAYZ/R5d6fc4TL9FVxH9e0RVH4GIiBwp2q4IRESkGyUCEZEoFzWJwMwuM7P1ZrbJzO4KdzzhYmaFZrbYzNaYWYmZfTLcMQ0GZhZrZm+b2V/CHUu4mVmmmT1uZuvMbK2ZnRXumMLFzD4d/Hfyrpk9amZJ4Y4pFKIiEZhZLHAfcDkwFbjRzKaGN6qwaQM+6+5TgfnAHVH8W3T2SWBtuIMYJH4A/N3dJwOziNLfxczygU8Ac919OoEpd28Ib1ShERWJAJgHbHL3Le7eAjwGXB3mmMLC3fe4+1vB93UE/pHnhzeq8DKzAuAK4BfhjiXczCwDOAf4JYC7t7h7dViDCq84INnM4oAUYHeY4wmJaEkE+cDOTsulRHnlB2BmRcAc4J9hDiXcvg98HugIcxyDwTigAvhVsKnsF2aWGu6gwsHddwHfBXYAe4Aad38uvFGFRrQkAunGzIYBTwCfcvfacMcTLmb2PmCvu68IdyyDRBxwGvBTd58DHACisk/NzLIItByMA0YDqWZ2c3ijCo1oSQS7gMJOywXBdVHJzOIJJIGH3f1P4Y4nzBYAV5nZNgJNhheY2e/CG1JYlQKl7n7wKvFxAokhGl0EbHX3CndvBf4EvCfMMYVEtCSCZcBEMxtnZgkEOnyeDnNMYWFmRqD9d627fy/c8YSbu3/R3QvcvYjA/xcvuXtEnvX1hbuXATvNbFJw1YXAmjCGFE47gPlmlhL8d3MhEdpxHhfuAAaCu7eZ2Z1AMYGe/wfdvSTMYYXLAuAWYLWZrQyu+5K7Pxu+kGSQ+TjwcPCkaQtwW5jjCQt3/6eZPQ68RWC03dtE6KMm9IgJEZEoFy1NQyIichRKBCIiUU6JQEQkyikRiIhEOSUCEZEop0QgMoDM7Dw94VQGGyUCEZEop0Qg0gMzu9nM3jSzlWZ2f3C+gnozuzf4fPoXzWx4sOxsM3vDzFaZ2ZPBZ9RgZhPM7AUze8fM3jKzU4KHH9bpef8PB+9aFQkbJQKRbsxsCnA9sMDdZwPtwE1AKrDc3acBS4CvBXf5DfAFd58JrO60/mHgPnefReAZNXuC6+cAnyIwN8Z4And7i4RNVDxiQuQ4XQicDiwLnqwnA3sJPKb698EyvwP+FHx+f6a7Lwmu/zXwRzNLA/Ld/UkAd28CCB7vTXcvDS6vBIqA10L+rUSOQolA5EgG/Nrdv9hlpdl/dit3os9nae70vh39O5QwU9OQyJFeBD5oZiMAzCzbzMYS+PfywWCZDwOvuXsNsN/MFgbX3wIsCc7+Vmpm7w8eI9HMUgbyS4j0lc5ERLpx9zVm9hXgOTOLAVqBOwhM0jIvuG0vgX4EgH8Bfhas6Ds/rfMW4H4zuzt4jA8N4NcQ6TM9fVSkj8ys3t2HhTsOkf6mpiERkSinKwIRkSinKwIRkSinRCAiEuWUCEREopwSgYhIlFMiEBGJcv8fkjp81Z9jbKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(f1s)\n",
    "plt.plot(f1s_eval)\n",
    "plt.title('model f1 micro')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7615385536398468"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(f1s_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Регуляризация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для того, чтобы сократить переобучение (а часто и просто чтобы улучшить качество) используют BatchNorm, DropOut и регуляризацию. \n",
    "\n",
    "DropOut - самый простой метод, он просто с какой-то вероятностью зануляет числа во входной матрице (есть еще dropout который зануляет рандомно, но уже сразу целиком какую-то размерность; в нашем случае он может например во всех эмбедингах занулить 2 и 5 элементы. Такой дропаут в торче называется DropOut2d). Использовать его очень легко - ```drop = nn.Dropout(0.3); drop(input)``` (также как relu в предыдущих примерах, но с 1 параметром). Для сверток Dropout однако почти не используют. \n",
    "\n",
    "В свертках используют BatchNorm. BatchNorm нормализует числа внутри батча так, чтобы они лежали в каком-то фиксированном интервале (от до 1). Это помогает сделать модель более устойчивой к изменениями во входных данных (например, если какое-то значение вдруг слишком сильно увеличится, то батчнорм не позволит ему слишком сильно повлиять на последующие слои), а также просто ускоряет обучение. Технически батчнорм не очень простой (там еще есть обучаемые параметры), вот несколько видео, если вам интересно разобраться -  https://www.youtube.com/watch?v=dXB-KQYkzNU , https://www.youtube.com/watch?v=nUUqwaxLnWs\n",
    "\n",
    "Регуляризация - это ограничения на значения параметров в модели. В торче добавить L2 регуляризацию можно в оптимайзере, задав параметр weight_decay и указав веса какого слоя нужно регуляризировать. Скорее всего вам это не пригодится, но полезно знать про такую возможность."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Добавим батчнорм и регуляризацию."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        self.VOCAB_SIZE = len(word2id)\n",
    "        self.EMB_DIM = 100\n",
    "        self.KERNEL_SIZE = 5\n",
    "        self.MAX_LEN = 200\n",
    "        self.IN_CHANNELS = self.EMB_DIM\n",
    "        self.OUT_CHANNELS = 128\n",
    "        self.STRIDE = 2\n",
    "        self.STRIDE_POOL = 2\n",
    "        self.KERNEL_SIZE_POOL = 2\n",
    "        self.HIDDEN_FC = 64\n",
    "        \n",
    "        \n",
    "        self.embedding = nn.Embedding(self.VOCAB_SIZE, self.EMB_DIM)\n",
    "        self.conv = nn.Conv1d(self.IN_CHANNELS, \n",
    "                              self.OUT_CHANNELS, \n",
    "                              kernel_size=self.KERNEL_SIZE, \n",
    "                              stride=self.STRIDE) \n",
    "        self.conv_length = int((((self.MAX_LEN-(self.KERNEL_SIZE-1)-1)/self.STRIDE)+1))\n",
    "        \n",
    "        # инициализируем батчнорм\n",
    "        self.batch_norm = nn.BatchNorm1d(self.conv_length)\n",
    "\n",
    "        \n",
    "        self.max_pool = nn.MaxPool1d(kernel_size=self.KERNEL_SIZE_POOL, \n",
    "                                     stride=self.STRIDE_POOL) \n",
    "        self.pool_length = int((((self.conv_length-(self.KERNEL_SIZE_POOL-1)-1)/self.STRIDE_POOL)+1))\n",
    "        \n",
    "        self.relu = torch.nn.LeakyReLU()\n",
    "        \n",
    "\n",
    "        self.fc = nn.Linear(self.OUT_CHANNELS * self.pool_length, \n",
    "                            self.HIDDEN_FC)\n",
    "        self.fc_1 = nn.Linear(self.HIDDEN_FC, output_dim)\n",
    "        self.act = nn.LogSoftmax(1) \n",
    "        \n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text).permute(0, 2, 1)\n",
    "        convoluted = self.conv(embedded)\n",
    "        # я не нашел совсем достоверной информации, но пару авторитетных мнений\n",
    "        # указывают на то, что батчнорм нужно ставить после свертки-пулинга-активации\n",
    "        # а не где-то внутри этой тройки\n",
    "        # то есть нужно держаться структуры свертка-*пулинг*-активация-батчнорм (пулинг опциональный)\n",
    "        pooled = self.batch_norm(self.relu(self.max_pool(convoluted)))\n",
    "        hidden = torch.flatten(pooled, 1, -1)\n",
    "        hidden = self.relu(self.fc(hidden))\n",
    "        dense_outputs=self.fc_1(hidden) \n",
    "        outputs=self.act(dense_outputs)\n",
    "        \n",
    "        return outputs\n",
    "\n",
    "\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(label2id))\n",
    "\n",
    "\n",
    "optimizer = optim.Adam([{'params':model.fc_1.parameters()},\n",
    "                        {'params':model.conv.parameters()},\n",
    "                        {'params':model.embedding.parameters()},\n",
    "                        {'params': model.fc.parameters(), 'weight_decay':0.1}],  # указываем в этом слое L2 регуляризацию\n",
    "                                                                                 # можно пробовать значения как в sklearn\n",
    "                                                                                 # в параметре C в логреге\n",
    "                       \n",
    "                         lr=0.001) # лернирг рейт тоже можно отдельно указать выше, но мы используем 1 на все слои\n",
    "criterion = nn.CrossEntropyLoss() \n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 2.6197411823272705; F1 measure: 0.1808203125\n",
      "Loss: 2.4586977422237397; F1 measure: 0.19806640625\n",
      "Loss: 2.373960111141205; F1 measure: 0.20907552083333333\n",
      "Loss: 2.293777464926243; F1 measure: 0.235419921875\n",
      "Loss: 2.1999969239234924; F1 measure: 0.2706328125\n",
      "Eval -  0.46314655172413793\n",
      "1\n",
      "Loss: 1.5232779133319854; F1 measure: 0.5215234375\n",
      "Loss: 1.4571399873495101; F1 measure: 0.54337890625\n",
      "Loss: 1.4014333546161652; F1 measure: 0.5609895833333334\n",
      "Loss: 1.355219143331051; F1 measure: 0.57517578125\n",
      "Loss: 1.313402940273285; F1 measure: 0.588875\n",
      "Eval -  0.6512631704980844\n",
      "2\n",
      "Loss: 1.0344296985864638; F1 measure: 0.6799609375\n",
      "Loss: 1.0047901949286462; F1 measure: 0.68927734375\n",
      "Loss: 0.9862580219904582; F1 measure: 0.6943098958333334\n",
      "Loss: 0.9652436164021492; F1 measure: 0.701083984375\n",
      "Loss: 0.9504220184087754; F1 measure: 0.705734375\n",
      "Eval -  0.7233494971264368\n",
      "3\n",
      "Loss: 0.8189972084760666; F1 measure: 0.74671875\n",
      "Loss: 0.812115375995636; F1 measure: 0.7473046875\n",
      "Loss: 0.8023925479253133; F1 measure: 0.7495442708333333\n",
      "Loss: 0.7957931512594223; F1 measure: 0.752353515625\n",
      "Loss: 0.7878573575019836; F1 measure: 0.7547265625\n",
      "Eval -  0.7482405411877395\n",
      "4\n",
      "Loss: 0.7163863027095795; F1 measure: 0.7746484375\n",
      "Loss: 0.7063124039769173; F1 measure: 0.77802734375\n",
      "Loss: 0.70201800207297; F1 measure: 0.7785416666666667\n",
      "Loss: 0.6974907685816288; F1 measure: 0.780703125\n",
      "Loss: 0.6953367521762848; F1 measure: 0.7814453125\n",
      "Eval -  0.7612601772030652\n",
      "5\n",
      "Loss: 0.6381947246193885; F1 measure: 0.801328125\n",
      "Loss: 0.6351038740575313; F1 measure: 0.8006640625\n",
      "Loss: 0.6329314880569776; F1 measure: 0.8010026041666667\n",
      "Loss: 0.6318752785772085; F1 measure: 0.80125\n",
      "Loss: 0.6284054705500602; F1 measure: 0.801921875\n",
      "Eval -  0.7726394875478927\n",
      "6\n",
      "Loss: 0.583585424721241; F1 measure: 0.8152734375\n",
      "Loss: 0.5796784365177154; F1 measure: 0.81775390625\n",
      "Loss: 0.5782570865750313; F1 measure: 0.8176432291666667\n",
      "Loss: 0.5759222892671824; F1 measure: 0.81875\n",
      "Loss: 0.5757328113317489; F1 measure: 0.8190625\n",
      "Eval -  0.7789146312260536\n",
      "7\n",
      "Loss: 0.5341010338068009; F1 measure: 0.831484375\n",
      "Loss: 0.5356463412940502; F1 measure: 0.8300390625\n",
      "Loss: 0.5386197704076767; F1 measure: 0.8299479166666667\n",
      "Loss: 0.5349517285823822; F1 measure: 0.83123046875\n",
      "Loss: 0.5324598643779754; F1 measure: 0.832328125\n",
      "Eval -  0.7886871408045977\n",
      "8\n",
      "Loss: 0.48952979385852813; F1 measure: 0.8453515625\n",
      "Loss: 0.4932489763200283; F1 measure: 0.8452734375\n",
      "Loss: 0.4943637558817863; F1 measure: 0.8448307291666667\n",
      "Loss: 0.49427898682653904; F1 measure: 0.8447265625\n",
      "Loss: 0.4922732102274895; F1 measure: 0.8455859375\n",
      "Eval -  0.7920222701149425\n",
      "9\n",
      "Loss: 0.46336703181266786; F1 measure: 0.85640625\n",
      "Loss: 0.46282783165574076; F1 measure: 0.85646484375\n",
      "Loss: 0.4612191939353943; F1 measure: 0.8561067708333333\n",
      "Loss: 0.4582536684721708; F1 measure: 0.8560546875\n",
      "Loss: 0.4556128804087639; F1 measure: 0.856703125\n",
      "Eval -  0.7923473419540229\n"
     ]
    }
   ],
   "source": [
    "f1s = []\n",
    "f1s_eval = []\n",
    "\n",
    "for i in range(10):\n",
    "    print(i)\n",
    "    f1s.append(train(model, training_generator, optimizer, criterion, 100))\n",
    "    evl = evaluate(model, valid_generator, criterion)\n",
    "    print('Eval - ', evl)\n",
    "    f1s_eval.append(evl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvG0lEQVR4nO3deXxV1bn/8c+TeSBzgEASCAgyT4qIRZzHWrXV1qHq7+rtldte7fxra4fbwdv25+3trZ1sq22tHRzaaq3a2sYJUStWQBEM80yAhCSQiczJ8/vjHCAJgQTIyUnO+b5fr7zO2Xuvvc9zjrKevddaey9zd0REJHrFhDsAEREJLyUCEZEop0QgIhLllAhERKKcEoGISJRTIhARiXJKBBKVzOwhM/tmH8tuM7OLjrIt2cyeMbMaM/tj/0Z5xGeNMbN6M4sN5edI9FEiEDk5HwRGAjnu/iEzG2VmT5vZbjNzMyvqrw9y9x3uPszd2/vrmCKgRCByssYCG9y9LbjcAfwduDZ8IXVlZnHhjkEGNyUCGbSCTTKfM7NVZnbAzH5pZiPN7G9mVmdmL5hZVqfyV5lZiZlVm9nLZjal07Y5ZvZWcL/fA0ndPut9ZrYyuO/rZjazD/F9A/gqcH2wyeYj7l7u7j8BlvX3dzSzouBVRlxwOdvMfhW8+thvZn8Orj/PzErN7AtmVgb8yswSzez7wbK7g+8T+xKjRD4lAhnsrgUuBk4FrgT+BnwJGE7g/99PAJjZqcCjwKeC254FnjGzBDNLAP4M/BbIBv5IpzN2M5sDPAj8O5AD3A883VtF6e5fA74N/D7YZPPLUH7HHvwWSAGmASOAezttyyPwXccCi4AvA/OB2cAsYB7wlROMVyKMEoEMdj8KnmXvAl4F/unub7t7E/AkMCdY7nrgr+7+vLu3At8FkoH3EKgA44Hvu3uruz9O1zP2RcD97v5Pd293918DzcH9BkJfv+MhZjYKuBz4qLvvD36vJZ2KdABfc/dmd28EbgLudve97l4BfAO4JdRfTIYGtR3KYFfe6X1jD8vDgu9HA9sPbnD3DjPbCeQD7cAu7/qExe2d3o8F/sXMPt5pXULwmAOhr9+xs0Jgn7vvP8oxK4KJ5KAuv0/w/UB9PxnkdEUgkWI3gQodADMzApXlLmAPkB9cd9CYTu93At9y98xOfynu/uhABH6CdgLZZpZ5lO3dHyvc5fch8P13hyAuGYKUCCRS/AG4wswuNLN44LMEmndeB5YCbcAnzCzezK4h0EZ+0M+Bj5rZmRaQamZXmFnaiQRiZknAwf6FxOByv3L3PQT6En5iZlnB73XOMXZ5FPiKmQ03s1wCndy/6++4ZGhSIpCI4O7rgZuBHwGVBDpdr3T3FndvAa4BbgX2EehP+FOnfZcDtwM/BvYDm4JlT1QjUB98vy64HAq3AK3Bz9hLoKP8aL4JLAdWAauBt4LrRDBNTCMiEt10RSAiEuWUCEREopwSgYhIlFMiEBGJckPuhrLc3FwvKioKdxgiIkPKihUrKt19eE/bhlwiKCoqYvny5eEOQ0RkSDGz7UfbpqYhEZEop0QgIhLllAhERKLckOsj6ElrayulpaU0NTX1XngIS0pKoqCggPj4+HCHIiIRJCISQWlpKWlpaRQVFdH1AZORw92pqqqitLSUcePGhTscEYkgEdE01NTURE5OTsQmAQAzIycnJ+KvekRk4EVEIgAiOgkcFA3fUUQGXkQ0DYmIRJq6plbKaprYU9N06PWCySOYUZDR75+lRNAPqqureeSRR/iP//iP49rvve99L4888giZmZmhCUxEBh13p7apjT01jV0q+bLg8sF19c1tXfYzg5xhCUoEg1V1dTU/+clPjkgEbW1txMUd/Sd+9tlnQx2aiAwgd6e6oTVQmdceruh3V3ddbmhp77KfGYxISyQvI5kJw4excGIuozKSyMtIDrymJzEyPYmEuNC05isR9IO77rqLzZs3M3v2bOLj40lKSiIrK4t169axYcMG3v/+97Nz506ampr45Cc/yaJFi4DDj8uor6/n8ssv5+yzz+b1118nPz+fp556iuTk5DB/MxE5yN3Zd6Cl01l7tzP62ib21DTS1NrRZb8Yg5HpSeRlJDE5L43zJ40IVvJJjMpIYlRGMsPTEomPDV+XbcQlgm88U8Ka3bX9esypo9P52pXTjrr9nnvu4d1332XlypW8/PLLXHHFFbz77ruHhnk++OCDZGdn09jYyBlnnMG1115LTk5Ol2Ns3LiRRx99lJ///Odcd911PPHEE9x88839+j1E5NhqGlrZvu8A26sa2LGvge1VB9ixryF4Rt9ES1vXSj42xsgLVvJTR6dz0ZQRh8/iM5IYnZFM7rAE4sJYyfdFxCWCwWDevHldxvr/8Ic/5MknnwRg586dbNy48YhEMG7cOGbPng3A6aefzrZt2wYqXJGo0dHhlNU2BSv6QIW/fV8DO/c1sL2qgZrG1i7lc4clMCY7hVmFmVze6Sw+LyOZ0RlJ5AxLJDZm6I/mi7hEcKwz94GSmpp66P3LL7/MCy+8wNKlS0lJSeG8887r8V6AxMTEQ+9jY2NpbAzVfOcika2ptZ3S/YGKvfOZ/fZ9DZTua6Sl/fBZfWyMUZCVzJjsFN43cxRjc1IYk50afE0hNTHiqsgeRce3DLG0tDTq6up63FZTU0NWVhYpKSmsW7eON954Y4CjE4k81Q0th87md1Qd6PS+gbLaridaqQmxjMlJ5dQRaVw8ZSSF2SmMzUlhbHYqozOTBn2zzUBQIugHOTk5LFiwgOnTp5OcnMzIkSMPbbvsssv42c9+xpQpU5g0aRLz588PY6QiQ0P7oSacA+zoVMlv3xdYrm3qOrRyRFoiY7JTeM+EHMYePKMPntXnpCboZsxemLuHO4bjMnfuXO8+Mc3atWuZMmVKmCIaWNH0XSWyNba0s/NQE06gU3ZHsMIv3d+1CSc+1ijIClTsY4Jn9IHXVAqzk0lJ0Dltb8xshbvP7Wmbfj0RCYmDwy0Pnc0H2+sPdtLurWvuUj4tMY4xOSlMHpXGJdPyDlX2Y7JTGJ2ZHBGdsoOVEoGInLC29g52Vzd1GXK5o9NInO53x+alJzEmJ4VzTx3O2JyUYHt9KmOzU8hMiVcTTpgoEYjIMdU3t7Gj03DLg00426sa2FXdSHvH4eblhLgYCrOSGZuTypnjsg8144zNSaEgK4Wk+NgwfhM5GiUCEcHdKd3fyLqyOtaX1bJpb/2hJp2qAy1dymamxDM2O4WZBRlcOWsUY7NTD3XM5qUnEaMmnCFHiUAkytQ0trK+rI51ZbWsK6tj3Z5aNpTXd2nGGZ2RxNicVC6eOpIxwaGWY7IDI3EykjVDXqRRIhCJUC1tHWyprGd9WR1r9wTO9NeX1bG75vA4+/SkOCaPSuea0/KZlJfG5Lx0Th05jLQkVfbRRIkgDIYNG0Z9fX24w5AI4R4Yc79uT13gDD9Y4W+uqKe1PdB+Hx9rnDJ8GPPGZTMpL53JeWlMHpVGXnqSOmhFiUBkKKlvbjvUrBN4DTTtdL7BanRGEpPy0jh/8ohAhZ+Xzrjc1JA9wliGvpAmAjO7DPgBEAv8wt3v6bZ9DPBrIDNY5i53H3IP6b/rrrsoLCzkjjvuAODrX/86cXFxLF68mP3799Pa2so3v/lNrr766jBHKkNFW3sH26oOBCv6QIW/vryWnfsOP4NqWGIck/LSeN+s0UzJS2NSXjqTRqaRkaJmHTk+Ibuz2MxigQ3AxUApsAy40d3XdCrzAPC2u//UzKYCz7p70bGO2+udxX+7C8pW9+M3AfJmwOX3HHXz22+/zac+9SmWLFkCwNSpUykuLiYjI4P09HQqKyuZP38+GzduxMxOqmlIdxZHnoaWNtbsrmX1rhre3VXLurJaNu6tP/TI49gYY3xuarANP3CGPykvjYKsZDXrSJ+F687iecAmd98SDOIx4GpgTacyDqQH32cAu0MYT8jMmTOHvXv3snv3bioqKsjKyiIvL49Pf/rTvPLKK8TExLBr1y7Ky8vJy8sLd7gSRo0t7azZU8vq0mpW76pl9a5qNu2t5+BQ/OFpiUwZlc6CCblMGhloxz9l+DCNv5eQCmUiyAd2dlouBc7sVubrwHNm9nEgFbiopwOZ2SJgEcCYMWOO/anHOHMPpQ996EM8/vjjlJWVcf311/Pwww9TUVHBihUriI+Pp6ioqMfHT0vkamptZ+2ewJn+6tIaVu+qYePe+kM3YOUOS2RmQQaXTx/FjPwMZhRkMDI9KcxRSzQKd2fxjcBD7v6/ZnYW8Fszm+7uXaYBcvcHgAcg0DQUhjh7df3113P77bdTWVnJkiVL+MMf/sCIESOIj49n8eLFbN++PdwhSgg1tbazrqwuWOkHzvY3lNcdqvRzUgOTjl8ydSTT8zOYWZDJyPRENe0MRh3t0NoIbU2Bv9Ym6DjYGe/g3sPr0bYF18NR9utlW/djD58EGQX9/pVDmQh2AYWdlguC6zr7CHAZgLsvNbMkIBfYG8K4QmLatGnU1dWRn5/PqFGjuOmmm7jyyiuZMWMGc+fOZfLkyeEOUfpJc1s76w9V+jWsKq1hQ3kdbcFKPzs1gen5GVw4eQQzCjKYkZ/BqAwN0zwh7a2HK+O2xuDrwQq6sdNrc6ftweVjbe9ynG77dLT2Hle4XPE9OOMj/X7YUCaCZcBEMxtHIAHcAHy4W5kdwIXAQ2Y2BUgCKkIYU0itXn24kzo3N5elS5f2WE73EAwdLW0dbCivY1WwaWf1rmrWl9UdGp+fmRLPjPwMFk0az8yCDKbnZ5CfqU7cXjXXQ3051O2BurJOr2Vdl1sPnPhnxCZAXDLEJUJ8UuD9oddkSMmGuKTA+7jEbtuTAtsObo852EdjYNbzKxx926H/HyxY9Gjbejl21uEpcPtTyBKBu7eZ2Z1AMYGhoQ+6e4mZ3Q0sd/engc8CPzezTxO4RrrVh9oECRIxWtsDlf7qQ5V+Dev21B16Ln56UhwzCzL5yNmBSn9GfoZG7nTX0hCoxI+o5Mu7VvYtPczoF5cEaaMgLQ9GzYRTL4XkrE4VcrfKPC4xWIknHX49VC6pU+UtvQlpH0HwnoBnu637aqf3a4AFoYxB5Gia29pZsX0/r22s5PXNVazZU3toyGZaUhwz8jO47ewiZuRnMDM/k8LsKK70WxsPV+L1ZUc5iy+D5poj941NDFTuaaNg5DSYcOHh5bQ8GJYXeE3K6HR2LAMp3J3F/cbdI/4fqS6WTo67s6G8nlc3VvDqxkre3LqPxtZ24mKM2YWZ3PqeokBHbn4GY7JTIvcpmq2N0FgNTTXQFHztvtxQ1fVMvqn6yOPEJhyu0IdPgvHnBZfzulb0SZmq4Ae5iEgESUlJVFVVkZOTE7HJwN2pqqoiKUnDC4/H3tomXttUyWsbK3ltU+WhWbHGD0/lurkFLJw4nDPHZw+th6x1tENzbe+V+dGW25uPcXAgYVigSSYtD3InwLiFMGzk4Yr94Gtylir4CBERiaCgoIDS0lIqKoZsP3OfJCUlUVDQ/0PHIkljSzv/3Fp1qOJfVxZoi85OTWDBhFwWTshlwcRc8jOTwxxpkDs07IOanVBTGjj7bqwOVt7VXSvzxprA++ZaDg077InFQnJmoKklKfiaUdB1ufP25Mzg+kxISofYIZQUpV9ERCKIj49n3LjQ9KbL4NbR4ZTsruWVjRW8trGSFdv309LeQUJcDGcUZfGFyyazcGIuU0elh6epp7UJancFKvlDfzu7Lrc1HrlffGrXyjq9AEZO76UyD75PSNWZuhyXiEgEEl1K9zfw2sZKXt1UyeubKtnfEBj3PWVUOrcuKOLsCbmcUZRNckKIR424w4HKIyv2zssHerglZlhe4Ax95LTAyJiMwsByRgGkjw40ueisXAaQEoEMenVNrSzdXMVrmyp5dWMlWysDY8tHpidyweSRLJyYy4IJuQxPS+zfD25thJpdx67ou7e3x6ccrtjzZnSt5A9W9HH9HKfISVIikEGnrb2Dd0qreWVDoJ1/5c5q2juclIRY5o/P4Zb5Y1k4MZcJI4ad3OCA5nrYv+3wX/dKvqGy2w4W6CjNKIBRs2DyFUdW9OpAlSFIiUDCzt3ZWnng0Bn/G5urqGtuI8ZgRkEmHzv3FM6emMtpY7KOb3KVjo5A5+uhyn5r14r/QLfBBQnDDlfso+cEK/dOFX3aKIhL6LfvLTJYKBFI2GyvOsAvXt3KS+v2sqs60GFamJ3M+2aN5pyJubznlNzeJ1lpaehauXep7Ld3bbqxmECFnlUEk94beM0qguxxkDlWZ/MStZQIZMBtrqjnvsWbeGrlbuJijPMnjeBj553Cwom5jM1J7VrYPfC4gn1be67w68u7lk9Ig+yiwA1Op17WtbLPKFQnrEgPlAhkwGwor+PHL23imVW7SYqL5V8XFHH7OeMZkeRQvQMqX4WN27pV9tu7Da+0w2f1Ey8OPIQrqyjwmj1OZ/UiJ0CJQEJuze5afrx4I8+uLiM1IZbPzR/GLXnbSdv9FPzytUAS6Cw+NVCp50yACRcdruiziiCzUKNuRPqZEoGEzOrSGn740kbeXrOB8xPX8fTY7UxrfofYt7cGCiRnQ9HZMOeWTmf2RZCaq7N6kQGkRCD97p0NW3j1+SdJ27OUu+LWckpSaWBDdTqMXQBnLoJx58CIqRBzHKOARCQklAjk5DXVwvbX2fPOczRvfJkZLVuYZU5rQhI29j1wyu2BB5flzYJY/S8nMtjoX6Ucv5YG2PkGbH0F3/oq7H4b83ayPZ5VMZOoHP9Rpp99JUljz9C4e5EhQIlAetfWDKXLYOsrsPXVwPuOVjosjg1xp/Jc65WsS5rNmQsv47qzTg39M35EpF8pEciR2lth99vBiv8V2PnPwCTfFoOPmsX2U2/loT2F/KG8gMyMTD52xSl8b24hSfFKACJDkRKBBCY6KVsdqPS3vQrbX4eW+sC2kdPh9NvoKFrIS40Tufe1ckq21FKYncxXr5nANacVHN9jH0Rk0FEiiFbVO2D93w5X/k3BuWZzJsLM6wOdu0ULaU/O4W/v7uFHf9/E+vKNjMtN5bsfmsXVs0cTH6sEIBIJlAiijTusfBj++n8Dd+xmjoUpV8K4c6FoIaSPAgJPAP3Lqj386KUlbK44wIQRw/jBDbO5YsYo4pQARCKKEkE0aa6Dv34WVv0+MI7/fd+HnFO6FGlt7+DPb+/ivsWb2FbVwOS8NH784TlcPn0UsZE6mbtIlFMiiBZ7VsHjt8G+LXD+l2HhZyHmcOduS1sHT7xVyn2LN1G6v5Fpo9P52c2nc8nUkeGZ4lFEBowSQaRzh+W/hL9/CVKy4V+eCTzWIaiptZ0/Lt/JT1/ezO6aJmYVZvKNq6ZxweQRJzfpi4gMGUoEkayxGp75BKx5CiZcDB/4WeA5PkBjSzuPvrmD+1/ZTHltM6ePzeL/XTuTcybmKgGIRJmQJgIzuwz4ARAL/MLd7+m2/V7g/OBiCjDC3TNDGVPU2LUC/ngb1O6Ci++Gsz5+6Lk+NQ2tfPBnr7Nxbz3zx2dz73WzOeuUHCUAkSgVskRgZrHAfcDFQCmwzMyedvc1B8u4+6c7lf84MCdU8UQNd3jjJ/D81yAtD277GxTOO7S5pa2Dj/5uBduqDvDgrXO5YPLIMAYrIoNBKK8I5gGb3H0LgJk9BlwNrDlK+RuBr4UwnsjXsA/+/DHY8HeY/D64+seBiVqC3J0vPbmapVuquPf6WUoCIgKENhHkAzs7LZcCZ/ZU0MzGAuOAl0IYT2TbvhSe+EhgQvbLvwPzFh3xTP/7Fm/i8RWlfPLCiXxgTkGYAhWRwWawdBbfADzu7u09bTSzRcAigDFjxgxkXINfRwf841546VuQOQY+8hyMPrKF7amVu/jucxv4wJx8PnXRxDAEKiKDVSgTwS6gsNNyQXBdT24A7jjagdz9AeABgLlz53p/BTjk1e+FPy2CLYth+rWBG8SS0o8otnzbPj73x1XMK8rmnmtnqFNYRLoIZSJYBkw0s3EEEsANwIe7FzKzyUAWsDSEsUSeLS8HkkBTDVz5Qzjt//Q4veO2ygPc/pvl5Gclc/8tp5MYpyeEikhXIUsE7t5mZncCxQSGjz7o7iVmdjew3N2fDha9AXjM3XWm3xftbbDkv+GV/4HcU+GWJ2HktB6LVje08K8PLQPgV7eeQVaqJokRkSOFtI/A3Z8Fnu227qvdlr8eyhgiSu1ueOLfYPs/YPbN8N7vQEJqj0Wb29pZ9NsVlO5v5OHbz6Qot+dyIiKDpbNYerPhOfjzR6G1CT5wP8y64ahF3Z0vPrGaN7fu4wc3zOaMouwBDFREhholgsGuvRVevBte/2FgkpgPPQS5xx7188MXN/Gnt3fxmYtP5erZ+QMTp4gMWUoEg9n+7YF7A0qXwdyPwKXfgvjkY+7y57d3ce8LG7jmtHw+fsGEAQpURIYyJYLBau0z8NQdgUdGfOghmPaBXnd5c+s+Pv/4KuaPz+aea2ZqmKiI9IkSwWDT1gzP/Se8eT+MPg0++CBkj+t1t62VB1j02+UUZCXzs5tP1zzCItJnSgSDSdXmwOQxe96Bs+6EC78Gcb0P+dx/oIXbfvUmMWb86rYzyEzRMFER6TslgsFi9ePwzKcgNg5ufAwmXd6n3QLDRJezu6aJR/7tTMbmaJioiBwfJYJwa2mAv98Fb/0aCufDB38JGX17IJy78/nHV7Fs235+eOMc5mqYqIicACWCcNq7LtAUtHdtYA7h874UuCLoo++/sJGnVu7mc5dO4qpZo0MYqIhEMiWCcHCHlQ/Ds58L3Bl88xMw4cLjOsQTK0r5wYsb+dDpBfzHeaeEKFARiQZKBAOtuR7++hlY9XsYdw5c8/PATGLH4Y0tVdz1p1WcNT6Hb31ATxMVkZOjRDDQnvx3WP8snP/lQHNQzPE9DXRzRT3//tsVjMlO0TBREekXSgQDadcKWPeXQBI49/PHvXtVfTO3/WoZcTHGr26dR0ZKfAiCFJFoo0QwkF76JqTkwPyPHfeuTa2Bp4mW1zbx6KL5jMlJCUGAIhKN1K4wULb9Aza/BGd/GhLTjmvXjg7nc4+vYsX2/XzvutmcNiar951ERPpIiWAguAeuBoblBR4ed5zufWEDz7yzm89fNokrZo4KQYAiEs2UCAbClsWw43U45/9CwvE16fxx+U5+9NImrp9byMfO1TBREel/SgSh5g4v/hdkFAbmFT4Or2+q5It/Ws2CCTl88wPTNUxUREJCiSDU1v8Ndr8VGCUUl9jn3Tbtreejv1vBuNxUfnLT6cTH6j+ViISGapdQ6uiAxd+C7PEw68N93q2qvpnbHnqThLgYHrz1DDKSNUxUREJHw0dDac2fofxduOYXfX6GUFNrO7f/Zjl7a5t5bNF8CrM1TFREQkuJIFTa22Dxt2H4FJh+TZ926ehwPvvHd3hrRzU/vek05miYqIgMACWCUFn9B6jaCNf9ts+Pkfjuc+v566o9fPHyyVw+Q8NERWRgqI8gFNpa4OV7YNQsmHJln3b5/bId/OTlzdw4bwyLzhkf4gBFRA7TFUEorPwdVG+HK/4X+jDk8x+bKvnyk++ycGIud189TcNERWRA6Yqgv7U2wZL/gcIzYcJFvRbfWF7HR3+3gvHDU7nvptM0TFREBlxIax0zu8zM1pvZJjO76yhlrjOzNWZWYmaPhDKeAbH8QajbDRd8pdergYq6Zm57aBmJcbE8eOsZpCdpmKiIDLyQNQ2ZWSxwH3AxUAosM7On3X1NpzITgS8CC9x9v5mNCFU8A6K5Hl77XmDCmXHnHLNoY0s7//ab5VTWN/P7RWdRkKVhoiISHqG8IpgHbHL3Le7eAjwGXN2tzO3Afe6+H8Dd94YwntB78wE4UAEX/Ocxi3V0OJ/5w0pWlVbz/evnMKswc2DiExHpQSgTQT6ws9NyaXBdZ6cCp5rZP8zsDTO7rKcDmdkiM1tuZssrKipCFO5JaqqBf/wAJl4KhfOOWfS/i9fxt3fL+PJ7p3DZ9OObplJEpL+Fu2cyDpgInAfcCPzczDK7F3L3B9x9rrvPHT58+MBG2FdLfwJN1XD+l45Z7E9vlXL/ki3cdOYYPnL2uIGJTUTkGEKZCHYBhZ2WC4LrOisFnnb3VnffCmwgkBiGloZ9sPQ+mHIVjJ59zKIPvLKF6fnpfOMqDRMVkcEhlIlgGTDRzMaZWQJwA/B0tzJ/JnA1gJnlEmgq2hLCmELjH9+Hlvperwa2Vx1gXVkd75+dT5yGiYrIIBGy2sjd24A7gWJgLfAHdy8xs7vN7KpgsWKgyszWAIuBz7l7VahiCom6cvjnAzDzOhgx5ZhFi0vKALh0mvoFRGTwCOmdxe7+LPBst3Vf7fTegc8E/4am174H7S1w7hd6LVpcUs6UUel6oqiIDCpqnzgZ1TsDN5DNuQlyjj2N5N66Jt7asZ9Lp40coOBERPpGieBkvPI/gddzPt9r0efXlOOuZiERGXxOKBGY2bD+DmTIqdoMb/8OTr8VMgt7LV5cUs6Y7BQm56WFPjYRkeNwolcEa3ovEuGWfAdiE2DhZ3stWtvUytLNlVw6baSGjIrIoHPUzmIzO1oHrgHRfUWwdx2s+j285+OQ1ntTz+J1e2ltdzULicigdKwrgm8DWUBat79hvewX+V7+NiSkwoJP9al4cUkZucMSOU1TT4rIIHSs4aNvAX929xXdN5jZv4UupEFuzzuw5qlAB3FqTq/Fm1rbeXl9BVfPzicmRs1CIjL4HOvMfhew3cw+2cO2uSGKZ/Bb/G1IyoSz7uhT8dc2VtLQ0q5hoyIyaB0rEUwFEoB/NbMsM8s++Ae0Dkx4g8zOZbDh77DgE5Cc2addikvKSEuM4z2n5IY2NhGRE3SspqH7gReB8cAKAp3EB3lwfXR56b8gJRfm/Xufire1d/DC2nLOnzyChLjo7lYRkcHrqLWTu//Q3acAD7r7eHcf1+kv+pLA1ldg6xJY+BlI7NugqWXb9rO/oVWjhURkUOv1NNXdPzYQgQxq7vDStyBtFMz91z7vVlxSRkJcDOdNGqRzKIiIEO3DQPtq04uw8w0453MQn9ynXdyd59eUs3BCLqmJIX22n4jISVEi6I17oG8gcwzMuaXPu727q5Zd1Y1qFhKRQU+JoDfr/gJ7VsK5d0FcQp93Ky4pI8bgwikjQhebiEg/UCI4lo72QN9AzgSYef1x7VpcUsYZRdnkDEsMUXAiIv1DieBYSp6EirVw3hchtu/t/Fsq6tm4t17NQiIyJCgRHE17W+Au4hHTYNo1x7VrcUk5AJfobmIRGQI0nOVo3nkU9m2GGx6BmOPLl8UlZUzPT6cgS1NSisjgpyuCnrQ1w5L/htFzYNJ7j2vXspomVu6s5tKpahYSkaFBVwQ9ees3ULMTrvw+HOdEMs+vKQPg0ulKBCIyNOiKoLvWRnjluzDmLDjlwuPevbiknHG5qUwcEd1z94jI0KFE0N2yX0B9GVzwn8d9NVDT0MobW6q4dFqepqQUkSFDiaCz5jp47V4Yfz4ULTju3V9cV05bh2vuAREZUpQIOvvnz6ChCi74ygntXlxSxsj0RGYVZPZvXCIiIaREcFDjfvjHj+DUy6Hg+Cdga2xpZ8mGCi6ZmqcpKUVkSAlpIjCzy8xsvZltMrO7eth+q5lVmNnK4F/45kJeeh8018AFXz6h3V/ZWEFTa4fuJhaRISdkw0fNLBa4D7gYKAWWmdnT7r6mW9Hfu/udoYqjTw5Uwhs/hWkfgLwZJ3SI4pIyMpLjOXN8dj8HJyISWqG8IpgHbHL3Le7eAjwGXB3Czztxr90LrQ2BZwqdgNb2Dl5cu5cLJ48gPlatbSIytISy1soHdnZaLg2u6+5aM1tlZo+bWWFPBzKzRWa23MyWV1RU9G+UtXsCQ0ZnXg/DJ53QId7cuo+axlYuUbOQiAxB4T59fQYocveZwPPAr3sq5O4PuPtcd587fHg/T/v46v9CRxuc+4UTPkRxSRlJ8TGce6qmpBSRoSeUiWAX0PkMvyC47hB3r3L35uDiL4DTQxjPkfZvhxUPBWYeyx53Qofo6HCeKynnnInDSU6I7d/4REQGQCgTwTJgopmNM7ME4Abg6c4FzGxUp8WrgLUhjOdIr3wHLCYwF/EJWrWrhrLaJo0WEpEhK2Sjhty9zczuBIqBWOBBdy8xs7uB5e7+NPAJM7sKaAP2AbeGKp4jVG6ClY/CvEWQ0VPXRd8Ul5QRG2OaklJEhqyQPn3U3Z8Fnu227qud3n8ROLGhOidryT0Qlwhnf/qkDlNcUsb88dlkpvR9PmMRkcEk3J3F4VG+BlY/HrgaSDvx5wJt2lvHlooDahYSkSEtOhPB4m9BYhos+ORJHebQlJSahEZEhrDoSwS734Z1f4Gz7oCUk7sLuLikjFmFmeRlJPVTcCIiAy/6EsFL34LkLJj/sZM6zO7qRlaV1uiR0yIy5EVXItjxBmx6PtAklJRxUod6riQ4JaX6B0RkiIueROAOL30TUkcEOolPUnFJORNGDOOU4ZqSUkSGtuhJBFuXwLZXYeFnISH1pA61/0ALb27bp2YhEYkI0ZMIanbB8Mlw+q0nfagX1pbT3uFqFhKRiBDSG8oGlTk3wawbIebkc19xSTmjM5KYkX9y/QwiIoNB9FwRQL8kgYaWNl7dWMEl0/Iw05SUIjL0RVci6AdL1lfQ3NbBJeofEJEIoURwnIpLyshKiWdekaakFJHIoERwHFraOnhx3V4unDKSOE1JKSIRQrXZcVi6pYq6pjaNFhKRiKJEcByKS8pISYhl4cTccIciItJvlAj6qKPDeX5NOeeeOpykeE1JKSKRQ4mgj97euZ+KumY1C4lIxFEi6KPiknLiYozzJ2tKShGJLEoEfeDuFJeUcdYpOWQkx4c7HBGRfqVE0Afry+vYXtWgZiERiUhKBH1Q/G45ZnDJVN1NLCKRR4mgD4pLyjhtTBYj0jUlpYhEHiWCXuzc18CaPbWae0BEIpYSQS+KNSWliEQ4JYJePFdSzuS8NMbmnNysZiIig5USwTFU1jezbPs+LtHVgIhEsJAmAjO7zMzWm9kmM7vrGOWuNTM3s7mhjOd4vbCmHHfUPyAiES1kicDMYoH7gMuBqcCNZja1h3JpwCeBf4YqlhNVXFJGQVYyU0elhzsUEZGQCeUVwTxgk7tvcfcW4DHg6h7K/Rfw30BTCGM5bnVNrfxjUxWXakpKEYlwoUwE+cDOTsulwXWHmNlpQKG7//VYBzKzRWa23MyWV1RU9H+kPXh5fQUt7R0aLSQiES9sncVmFgN8D/hsb2Xd/QF3n+vuc4cPHx764Ag0C+WkJnD62KwB+TwRkXAJZSLYBRR2Wi4IrjsoDZgOvGxm24D5wNODocO4ua2dl9dXcPHUkcTGqFlIRCJbKBPBMmCimY0zswTgBuDpgxvdvcbdc929yN2LgDeAq9x9eQhj6pPXN1VR36wpKUUkOoQsEbh7G3AnUAysBf7g7iVmdreZXRWqz+0PxSVlDEuM4z0TcsIdiohIyMWF8uDu/izwbLd1Xz1K2fNCGUtftQenpDxv0nAS4zQlpYhEPt1Z3M2K7fupOtCiZiERiRpKBN0Ul5SREBvDeZMGZnSSiEi4KRF0cnBKygUTckhL0pSUIhIdlAg6WbOnltL9jWoWEpGookTQSXFJOTEGF2lKShGJIkoEnTxXUsbcsdnkDksMdygiIgNGiSBoe9UB1pXVcYkeOS0iUUaJIEhTUopItFIiCCouKWfqqHQKs1PCHYqIyIBSIgD21jXx1o79uhoQkaikRAA8f3BKyunqHxCR6KNEQKBZaGxOCpNGpoU7FBGRARf1iaC2qZWlmys1JaWIRK2oTwSL1+2ltd25VMNGRSRKRX0iKC4pY3haInMKNSWliESnqE4ETa2Hp6SM0ZSUIhKlojoRvLaxkoaWdg0bFZGoFtWJoLikjLSkOM4arykpRSR6RW0iaGvv4IW15VwweQQJcVH7M4iIRG8iWLZtP/sbWtUsJCJRL2oTQXFJGYlxMZx7qqakFJHoFpWJwN15fk05CycOJzUxLtzhiIiEVVQmgnd31bKrulE3kYmIEKWJoLikjNgY46IpSgQiIlGbCOYVZZOVmhDuUEREwi7qEsGWino27q1Xs5CISFBIE4GZXWZm681sk5nd1cP2j5rZajNbaWavmdnUUMYDgUdOA1yiYaMiIkAIE4GZxQL3AZcDU4Ebe6joH3H3Ge4+G/gO8L1QxXNQcUkZMwsyGJ2ZHOqPEhEZEkJ5RTAP2OTuW9y9BXgMuLpzAXev7bSYCngI46GspomVO6t1E5mISCehHESfD+zstFwKnNm9kJndAXwGSAAu6OlAZrYIWAQwZsyYEw7ouTVlAOofEBHpJOydxe5+n7ufAnwB+MpRyjzg7nPdfe7w4Sd+J3BxSRnjh6cyYYSmpBQROSiUiWAXUNhpuSC47mgeA94fqmCqG1p4Y8s+NQuJiHQTykSwDJhoZuPMLAG4AXi6cwEzm9hp8QpgY6iCeXHtXto7XIlARKSbkPURuHubmd0JFAOxwIPuXmJmdwPL3f1p4E4zuwhoBfYD/xKqeNKT47l46khm5meE6iNERIYkcw/pQJ1+N3fuXF++fHm4wxARGVLMbIW7z+1pW9g7i0VEJLyUCEREopwSgYhIlFMiEBGJckoEIiJRTolARCTKKRGIiEQ5JQIRkSg35G4oM7MKYPsJ7p4LVPZjOEOdfo+u9Hscpt+iq0j4Pca6e49P7RxyieBkmNnyo91ZF430e3Sl3+Mw/RZdRfrvoaYhEZEop0QgIhLloi0RPBDuAAYZ/R5d6fc4TL9FVxH9e0RVH4GIiBwp2q4IRESkGyUCEZEoFzWJwMwuM7P1ZrbJzO4KdzzhYmaFZrbYzNaYWYmZfTLcMQ0GZhZrZm+b2V/CHUu4mVmmmT1uZuvMbK2ZnRXumMLFzD4d/Hfyrpk9amZJ4Y4pFKIiEZhZLHAfcDkwFbjRzKaGN6qwaQM+6+5TgfnAHVH8W3T2SWBtuIMYJH4A/N3dJwOziNLfxczygU8Ac919OoEpd28Ib1ShERWJAJgHbHL3Le7eAjwGXB3mmMLC3fe4+1vB93UE/pHnhzeq8DKzAuAK4BfhjiXczCwDOAf4JYC7t7h7dViDCq84INnM4oAUYHeY4wmJaEkE+cDOTsulRHnlB2BmRcAc4J9hDiXcvg98HugIcxyDwTigAvhVsKnsF2aWGu6gwsHddwHfBXYAe4Aad38uvFGFRrQkAunGzIYBTwCfcvfacMcTLmb2PmCvu68IdyyDRBxwGvBTd58DHACisk/NzLIItByMA0YDqWZ2c3ijCo1oSQS7gMJOywXBdVHJzOIJJIGH3f1P4Y4nzBYAV5nZNgJNhheY2e/CG1JYlQKl7n7wKvFxAokhGl0EbHX3CndvBf4EvCfMMYVEtCSCZcBEMxtnZgkEOnyeDnNMYWFmRqD9d627fy/c8YSbu3/R3QvcvYjA/xcvuXtEnvX1hbuXATvNbFJw1YXAmjCGFE47gPlmlhL8d3MhEdpxHhfuAAaCu7eZ2Z1AMYGe/wfdvSTMYYXLAuAWYLWZrQyu+5K7Pxu+kGSQ+TjwcPCkaQtwW5jjCQt3/6eZPQ68RWC03dtE6KMm9IgJEZEoFy1NQyIichRKBCIiUU6JQEQkyikRiIhEOSUCEZEop0QgMoDM7Dw94VQGGyUCEZEop0Qg0gMzu9nM3jSzlWZ2f3C+gnozuzf4fPoXzWx4sOxsM3vDzFaZ2ZPBZ9RgZhPM7AUze8fM3jKzU4KHH9bpef8PB+9aFQkbJQKRbsxsCnA9sMDdZwPtwE1AKrDc3acBS4CvBXf5DfAFd58JrO60/mHgPnefReAZNXuC6+cAnyIwN8Z4And7i4RNVDxiQuQ4XQicDiwLnqwnA3sJPKb698EyvwP+FHx+f6a7Lwmu/zXwRzNLA/Ld/UkAd28CCB7vTXcvDS6vBIqA10L+rUSOQolA5EgG/Nrdv9hlpdl/dit3os9nae70vh39O5QwU9OQyJFeBD5oZiMAzCzbzMYS+PfywWCZDwOvuXsNsN/MFgbX3wIsCc7+Vmpm7w8eI9HMUgbyS4j0lc5ERLpx9zVm9hXgOTOLAVqBOwhM0jIvuG0vgX4EgH8Bfhas6Ds/rfMW4H4zuzt4jA8N4NcQ6TM9fVSkj8ys3t2HhTsOkf6mpiERkSinKwIRkSinKwIRkSinRCAiEuWUCEREopwSgYhIlFMiEBGJcv8fkjp81Z9jbKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(f1s)\n",
    "plt.plot(f1s_eval)\n",
    "plt.title('model f1 micro')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7343342911877394"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(f1s_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Несколько слоев с паддингом"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Во всех сетках выше только 1 CNN слой и еще 1 доп. пулинг. Но рабочие сверточные сети это обычно десятки сверток и пулингов, которые наслаиваются друг на друга. Из когда выше видно, что собрать такую сетку будет непросто - будет очень много промежуточного когда для расчета размерностей. Чтобы этого избежать можно написать отдельный класс, который будет применять сверточный слой и сохранять длину последовательности. Для этого мы воспользуемся паддингом - добьем нулями исходную последовательность, чтобы на выходе получилась изначальная длина."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "# созданный класс в торче можно использовать просто как слой!\n",
    "\n",
    "class Conv_layer_with_pad(nn.Module):\n",
    "    \n",
    "    def __init__(self, in_channels, out_channels):\n",
    "        \n",
    "        super().__init__()          \n",
    "        self.KERNEL_SIZE = 7\n",
    "        self.IN_CHANNELS = in_channels\n",
    "        self.OUT_CHANNELS = 32\n",
    "        self.STRIDE = 1\n",
    "        self.conv = nn.Conv1d(self.IN_CHANNELS, \n",
    "                              self.OUT_CHANNELS, \n",
    "                              kernel_size=self.KERNEL_SIZE, \n",
    "                              stride=self.STRIDE, ) \n",
    "        \n",
    "        self.act = nn.LeakyReLU()\n",
    "        \n",
    "    def forward(self, embedded):\n",
    "        \"\"\"\n",
    "        Раньше мы расчитывали длину, которая получится, после применения сверточного слоя  по формуле:\n",
    "        OUTPUT_LENGTH = (((MAX_LEN-(KERNEL_SIZE-1)-1)/STRIDE)+1)\n",
    "        \n",
    "        \n",
    "        Теперь мы хотим на выходе получить длину, которая будет соответствовать длине изначальной последовательности.\n",
    "        Т.е. мы знаем OUTPUT_LENGTH и наша задача найти такой MAX_LEN, при котором он получается (остальные параметры фиксированы)\n",
    "        Перенесем все известные значения влево и получится вот такая формула\n",
    "        ((OUTPUT_LENGTH-1)*STRIDE)+1+(KERNEL_SIZE-1) = MAX_LEN\n",
    "        \n",
    "        Используя нужное значение MAX_LEN, нам нужно допаддить изначальную последовательность до этой длины\n",
    "        Т.к. мы будет падить с двух сторон, то нужно, чтобы получившееся число делилось на 2\n",
    "        \n",
    "        \n",
    "        \n",
    "        \"\"\"\n",
    "        MAX_LEN = embedded.size(2)\n",
    "        EXP_LEN = ((MAX_LEN-1)*self.STRIDE)+1+(self.KERNEL_SIZE-1)\n",
    "\n",
    "        # нужно чтобы все сходилось, иначе выдавать ошибку\n",
    "        assert not (EXP_LEN-MAX_LEN)%2, f\"Не ровный паддинг! {EXP_LEN}\"\n",
    "        \n",
    "        # допустим чтобы получить изначальную длину в 200\n",
    "        # нужно добавить нулей до 210\n",
    "        # тогда мы добавляем 5 нулей слева и 5 справа\n",
    "        # ассерт выше выкинул бы ошибку если бы у нас получалось например 209, \n",
    "        # то есть мы не смогли бы ровно западить с 2 сторон\n",
    "        padded = F.pad(embedded, (int((EXP_LEN-MAX_LEN)/2), \n",
    "                                                   int((EXP_LEN-MAX_LEN)/2)))\n",
    "        \n",
    "        convoluted = self.conv(padded)\n",
    "        \n",
    "        return self.act(convoluted)\n",
    "\n",
    "\n",
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        self.VOCAB_SIZE = len(word2id)\n",
    "        self.EMB_DIM = 100\n",
    "        self.KERNEL_SIZE = 7\n",
    "        self.MAX_LEN = 200\n",
    "        self.IN_CHANNELS = self.MAX_LEN\n",
    "        self.OUT_CHANNELS = 32\n",
    "        self.STRIDE = 1\n",
    "        self.HIDDEN_FC = 64\n",
    "        \n",
    "        \n",
    "        self.embedding = nn.Embedding(self.VOCAB_SIZE, self.EMB_DIM)\n",
    "        \n",
    "        # теперь слои удобно стакать\n",
    "        # нужно только указывать OUT_CHANNELS как IN_CHANNELS в следующем слое\n",
    "        self.conv = Conv_layer_with_pad(self.EMB_DIM, self.OUT_CHANNELS)\n",
    "        self.conv1 = Conv_layer_with_pad(self.OUT_CHANNELS, 16)\n",
    "        self.relu = nn.ReLU()\n",
    "        # если бы мы поставили еще 1 слой, то его IN_CHANNELS была бы 16 и тд\n",
    "\n",
    "        # длина сохраняется поэтому просто умножаем макс лен на OUT_CHANNELS последнего слоя\n",
    "        self.flattened_size = int(self.MAX_LEN*16)\n",
    "        \n",
    "        self.fc = nn.Linear(self.flattened_size, \n",
    "                            self.HIDDEN_FC)\n",
    "        self.fc_1 = nn.Linear(self.HIDDEN_FC, output_dim)\n",
    "        self.act = nn.LogSoftmax(1) \n",
    "        \n",
    "        \n",
    "    def forward(self, text):\n",
    "\n",
    "        embedded = self.embedding(text)\n",
    "        convoluted = self.relu(self.conv1(self.conv(embedded.permute(0, 2, 1))))\n",
    "        hidden = torch.flatten(convoluted, 1, -1)\n",
    "        hidden = torch.relu(self.fc(hidden))\n",
    "        dense_outputs=self.fc_1(hidden) \n",
    "        outputs=self.act(dense_outputs)\n",
    "\n",
    "        return outputs\n",
    "\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(label2id))\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss() \n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 2.6197411823272705; F1 measure: 0.1808203125\n",
      "Loss: 2.4586977422237397; F1 measure: 0.19806640625\n",
      "Loss: 2.373960111141205; F1 measure: 0.20907552083333333\n",
      "Loss: 2.293777464926243; F1 measure: 0.235419921875\n",
      "Loss: 2.1999969239234924; F1 measure: 0.2706328125\n",
      "Eval -  0.46314655172413793\n",
      "1\n",
      "Loss: 1.5232779133319854; F1 measure: 0.5215234375\n",
      "Loss: 1.4571399873495101; F1 measure: 0.54337890625\n",
      "Loss: 1.4014333546161652; F1 measure: 0.5609895833333334\n",
      "Loss: 1.355219143331051; F1 measure: 0.57517578125\n",
      "Loss: 1.313402940273285; F1 measure: 0.588875\n",
      "Eval -  0.6512631704980844\n",
      "2\n",
      "Loss: 1.0344296985864638; F1 measure: 0.6799609375\n",
      "Loss: 1.0047901949286462; F1 measure: 0.68927734375\n",
      "Loss: 0.9862580219904582; F1 measure: 0.6943098958333334\n",
      "Loss: 0.9652436164021492; F1 measure: 0.701083984375\n",
      "Loss: 0.9504220184087754; F1 measure: 0.705734375\n",
      "Eval -  0.7233494971264368\n",
      "3\n",
      "Loss: 0.8189972084760666; F1 measure: 0.74671875\n",
      "Loss: 0.812115375995636; F1 measure: 0.7473046875\n",
      "Loss: 0.8023925479253133; F1 measure: 0.7495442708333333\n",
      "Loss: 0.7957931512594223; F1 measure: 0.752353515625\n",
      "Loss: 0.7878573575019836; F1 measure: 0.7547265625\n",
      "Eval -  0.7482405411877395\n",
      "4\n",
      "Loss: 0.7163863027095795; F1 measure: 0.7746484375\n",
      "Loss: 0.7063124039769173; F1 measure: 0.77802734375\n",
      "Loss: 0.70201800207297; F1 measure: 0.7785416666666667\n",
      "Loss: 0.6974907685816288; F1 measure: 0.780703125\n",
      "Loss: 0.6953367521762848; F1 measure: 0.7814453125\n",
      "Eval -  0.7612601772030652\n",
      "5\n",
      "Loss: 0.6381947246193885; F1 measure: 0.801328125\n",
      "Loss: 0.6351038740575313; F1 measure: 0.8006640625\n",
      "Loss: 0.6329314880569776; F1 measure: 0.8010026041666667\n",
      "Loss: 0.6318752785772085; F1 measure: 0.80125\n",
      "Loss: 0.6284054705500602; F1 measure: 0.801921875\n",
      "Eval -  0.7726394875478927\n",
      "6\n",
      "Loss: 0.583585424721241; F1 measure: 0.8152734375\n",
      "Loss: 0.5796784365177154; F1 measure: 0.81775390625\n",
      "Loss: 0.5782570865750313; F1 measure: 0.8176432291666667\n",
      "Loss: 0.5759222892671824; F1 measure: 0.81875\n",
      "Loss: 0.5757328113317489; F1 measure: 0.8190625\n",
      "Eval -  0.7789146312260536\n",
      "7\n",
      "Loss: 0.5341010338068009; F1 measure: 0.831484375\n",
      "Loss: 0.5356463412940502; F1 measure: 0.8300390625\n",
      "Loss: 0.5386197704076767; F1 measure: 0.8299479166666667\n",
      "Loss: 0.5349517285823822; F1 measure: 0.83123046875\n",
      "Loss: 0.5324598643779754; F1 measure: 0.832328125\n",
      "Eval -  0.7886871408045977\n",
      "8\n",
      "Loss: 0.48952979385852813; F1 measure: 0.8453515625\n",
      "Loss: 0.4932489763200283; F1 measure: 0.8452734375\n",
      "Loss: 0.4943637558817863; F1 measure: 0.8448307291666667\n",
      "Loss: 0.49427898682653904; F1 measure: 0.8447265625\n",
      "Loss: 0.4922732102274895; F1 measure: 0.8455859375\n",
      "Eval -  0.7920222701149425\n",
      "9\n",
      "Loss: 0.46336703181266786; F1 measure: 0.85640625\n",
      "Loss: 0.46282783165574076; F1 measure: 0.85646484375\n",
      "Loss: 0.4612191939353943; F1 measure: 0.8561067708333333\n",
      "Loss: 0.4582536684721708; F1 measure: 0.8560546875\n",
      "Loss: 0.4556128804087639; F1 measure: 0.856703125\n",
      "Eval -  0.7923473419540229\n"
     ]
    }
   ],
   "source": [
    "f1s = []\n",
    "f1s_eval = []\n",
    "\n",
    "for i in range(10):\n",
    "    print(i)\n",
    "    f1s.append(train(model, training_generator, optimizer, criterion, 100))\n",
    "    evl = evaluate(model, valid_generator, criterion)\n",
    "    print('Eval - ', evl)\n",
    "    f1s_eval.append(evl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvG0lEQVR4nO3deXxV1bn/8c+TeSBzgEASCAgyT4qIRZzHWrXV1qHq7+rtldte7fxra4fbwdv25+3trZ1sq22tHRzaaq3a2sYJUStWQBEM80yAhCSQiczJ8/vjHCAJgQTIyUnO+b5fr7zO2Xuvvc9zjrKevddaey9zd0REJHrFhDsAEREJLyUCEZEop0QgIhLllAhERKKcEoGISJRTIhARiXJKBBKVzOwhM/tmH8tuM7OLjrIt2cyeMbMaM/tj/0Z5xGeNMbN6M4sN5edI9FEiEDk5HwRGAjnu/iEzG2VmT5vZbjNzMyvqrw9y9x3uPszd2/vrmCKgRCByssYCG9y9LbjcAfwduDZ8IXVlZnHhjkEGNyUCGbSCTTKfM7NVZnbAzH5pZiPN7G9mVmdmL5hZVqfyV5lZiZlVm9nLZjal07Y5ZvZWcL/fA0ndPut9ZrYyuO/rZjazD/F9A/gqcH2wyeYj7l7u7j8BlvX3dzSzouBVRlxwOdvMfhW8+thvZn8Orj/PzErN7AtmVgb8yswSzez7wbK7g+8T+xKjRD4lAhnsrgUuBk4FrgT+BnwJGE7g/99PAJjZqcCjwKeC254FnjGzBDNLAP4M/BbIBv5IpzN2M5sDPAj8O5AD3A883VtF6e5fA74N/D7YZPPLUH7HHvwWSAGmASOAezttyyPwXccCi4AvA/OB2cAsYB7wlROMVyKMEoEMdj8KnmXvAl4F/unub7t7E/AkMCdY7nrgr+7+vLu3At8FkoH3EKgA44Hvu3uruz9O1zP2RcD97v5Pd293918DzcH9BkJfv+MhZjYKuBz4qLvvD36vJZ2KdABfc/dmd28EbgLudve97l4BfAO4JdRfTIYGtR3KYFfe6X1jD8vDgu9HA9sPbnD3DjPbCeQD7cAu7/qExe2d3o8F/sXMPt5pXULwmAOhr9+xs0Jgn7vvP8oxK4KJ5KAuv0/w/UB9PxnkdEUgkWI3gQodADMzApXlLmAPkB9cd9CYTu93At9y98xOfynu/uhABH6CdgLZZpZ5lO3dHyvc5fch8P13hyAuGYKUCCRS/AG4wswuNLN44LMEmndeB5YCbcAnzCzezK4h0EZ+0M+Bj5rZmRaQamZXmFnaiQRiZknAwf6FxOByv3L3PQT6En5iZlnB73XOMXZ5FPiKmQ03s1wCndy/6++4ZGhSIpCI4O7rgZuBHwGVBDpdr3T3FndvAa4BbgX2EehP+FOnfZcDtwM/BvYDm4JlT1QjUB98vy64HAq3AK3Bz9hLoKP8aL4JLAdWAauBt4LrRDBNTCMiEt10RSAiEuWUCEREopwSgYhIlFMiEBGJckPuhrLc3FwvKioKdxgiIkPKihUrKt19eE/bhlwiKCoqYvny5eEOQ0RkSDGz7UfbpqYhEZEop0QgIhLllAhERKLckOsj6ElrayulpaU0NTX1XngIS0pKoqCggPj4+HCHIiIRJCISQWlpKWlpaRQVFdH1AZORw92pqqqitLSUcePGhTscEYkgEdE01NTURE5OTsQmAQAzIycnJ+KvekRk4EVEIgAiOgkcFA3fUUQGXkQ0DYmIRJq6plbKaprYU9N06PWCySOYUZDR75+lRNAPqqureeSRR/iP//iP49rvve99L4888giZmZmhCUxEBh13p7apjT01jV0q+bLg8sF19c1tXfYzg5xhCUoEg1V1dTU/+clPjkgEbW1txMUd/Sd+9tlnQx2aiAwgd6e6oTVQmdceruh3V3ddbmhp77KfGYxISyQvI5kJw4excGIuozKSyMtIDrymJzEyPYmEuNC05isR9IO77rqLzZs3M3v2bOLj40lKSiIrK4t169axYcMG3v/+97Nz506ampr45Cc/yaJFi4DDj8uor6/n8ssv5+yzz+b1118nPz+fp556iuTk5DB/MxE5yN3Zd6Cl01l7tzP62ib21DTS1NrRZb8Yg5HpSeRlJDE5L43zJ40IVvJJjMpIYlRGMsPTEomPDV+XbcQlgm88U8Ka3bX9esypo9P52pXTjrr9nnvu4d1332XlypW8/PLLXHHFFbz77ruHhnk++OCDZGdn09jYyBlnnMG1115LTk5Ol2Ns3LiRRx99lJ///Odcd911PPHEE9x88839+j1E5NhqGlrZvu8A26sa2LGvge1VB9ixryF4Rt9ES1vXSj42xsgLVvJTR6dz0ZQRh8/iM5IYnZFM7rAE4sJYyfdFxCWCwWDevHldxvr/8Ic/5MknnwRg586dbNy48YhEMG7cOGbPng3A6aefzrZt2wYqXJGo0dHhlNU2BSv6QIW/fV8DO/c1sL2qgZrG1i7lc4clMCY7hVmFmVze6Sw+LyOZ0RlJ5AxLJDZm6I/mi7hEcKwz94GSmpp66P3LL7/MCy+8wNKlS0lJSeG8887r8V6AxMTEQ+9jY2NpbAzVfOcika2ptZ3S/YGKvfOZ/fZ9DZTua6Sl/fBZfWyMUZCVzJjsFN43cxRjc1IYk50afE0hNTHiqsgeRce3DLG0tDTq6up63FZTU0NWVhYpKSmsW7eON954Y4CjE4k81Q0th87md1Qd6PS+gbLaridaqQmxjMlJ5dQRaVw8ZSSF2SmMzUlhbHYqozOTBn2zzUBQIugHOTk5LFiwgOnTp5OcnMzIkSMPbbvsssv42c9+xpQpU5g0aRLz588PY6QiQ0P7oSacA+zoVMlv3xdYrm3qOrRyRFoiY7JTeM+EHMYePKMPntXnpCboZsxemLuHO4bjMnfuXO8+Mc3atWuZMmVKmCIaWNH0XSWyNba0s/NQE06gU3ZHsMIv3d+1CSc+1ijIClTsY4Jn9IHXVAqzk0lJ0Dltb8xshbvP7Wmbfj0RCYmDwy0Pnc0H2+sPdtLurWvuUj4tMY4xOSlMHpXGJdPyDlX2Y7JTGJ2ZHBGdsoOVEoGInLC29g52Vzd1GXK5o9NInO53x+alJzEmJ4VzTx3O2JyUYHt9KmOzU8hMiVcTTpgoEYjIMdU3t7Gj03DLg00426sa2FXdSHvH4eblhLgYCrOSGZuTypnjsg8144zNSaEgK4Wk+NgwfhM5GiUCEcHdKd3fyLqyOtaX1bJpb/2hJp2qAy1dymamxDM2O4WZBRlcOWsUY7NTD3XM5qUnEaMmnCFHiUAkytQ0trK+rI51ZbWsK6tj3Z5aNpTXd2nGGZ2RxNicVC6eOpIxwaGWY7IDI3EykjVDXqRRIhCJUC1tHWyprGd9WR1r9wTO9NeX1bG75vA4+/SkOCaPSuea0/KZlJfG5Lx0Th05jLQkVfbRRIkgDIYNG0Z9fX24w5AI4R4Yc79uT13gDD9Y4W+uqKe1PdB+Hx9rnDJ8GPPGZTMpL53JeWlMHpVGXnqSOmhFiUBkKKlvbjvUrBN4DTTtdL7BanRGEpPy0jh/8ohAhZ+Xzrjc1JA9wliGvpAmAjO7DPgBEAv8wt3v6bZ9DPBrIDNY5i53H3IP6b/rrrsoLCzkjjvuAODrX/86cXFxLF68mP3799Pa2so3v/lNrr766jBHKkNFW3sH26oOBCv6QIW/vryWnfsOP4NqWGIck/LSeN+s0UzJS2NSXjqTRqaRkaJmHTk+Ibuz2MxigQ3AxUApsAy40d3XdCrzAPC2u//UzKYCz7p70bGO2+udxX+7C8pW9+M3AfJmwOX3HHXz22+/zac+9SmWLFkCwNSpUykuLiYjI4P09HQqKyuZP38+GzduxMxOqmlIdxZHnoaWNtbsrmX1rhre3VXLurJaNu6tP/TI49gYY3xuarANP3CGPykvjYKsZDXrSJ+F687iecAmd98SDOIx4GpgTacyDqQH32cAu0MYT8jMmTOHvXv3snv3bioqKsjKyiIvL49Pf/rTvPLKK8TExLBr1y7Ky8vJy8sLd7gSRo0t7azZU8vq0mpW76pl9a5qNu2t5+BQ/OFpiUwZlc6CCblMGhloxz9l+DCNv5eQCmUiyAd2dlouBc7sVubrwHNm9nEgFbiopwOZ2SJgEcCYMWOO/anHOHMPpQ996EM8/vjjlJWVcf311/Pwww9TUVHBihUriI+Pp6ioqMfHT0vkamptZ+2ewJn+6tIaVu+qYePe+kM3YOUOS2RmQQaXTx/FjPwMZhRkMDI9KcxRSzQKd2fxjcBD7v6/ZnYW8Fszm+7uXaYBcvcHgAcg0DQUhjh7df3113P77bdTWVnJkiVL+MMf/sCIESOIj49n8eLFbN++PdwhSgg1tbazrqwuWOkHzvY3lNcdqvRzUgOTjl8ydSTT8zOYWZDJyPRENe0MRh3t0NoIbU2Bv9Ym6DjYGe/g3sPr0bYF18NR9utlW/djD58EGQX9/pVDmQh2AYWdlguC6zr7CHAZgLsvNbMkIBfYG8K4QmLatGnU1dWRn5/PqFGjuOmmm7jyyiuZMWMGc+fOZfLkyeEOUfpJc1s76w9V+jWsKq1hQ3kdbcFKPzs1gen5GVw4eQQzCjKYkZ/BqAwN0zwh7a2HK+O2xuDrwQq6sdNrc6ftweVjbe9ynG77dLT2Hle4XPE9OOMj/X7YUCaCZcBEMxtHIAHcAHy4W5kdwIXAQ2Y2BUgCKkIYU0itXn24kzo3N5elS5f2WE73EAwdLW0dbCivY1WwaWf1rmrWl9UdGp+fmRLPjPwMFk0az8yCDKbnZ5CfqU7cXjXXQ3051O2BurJOr2Vdl1sPnPhnxCZAXDLEJUJ8UuD9oddkSMmGuKTA+7jEbtuTAtsObo852EdjYNbzKxx926H/HyxY9Gjbejl21uEpcPtTyBKBu7eZ2Z1AMYGhoQ+6e4mZ3Q0sd/engc8CPzezTxO4RrrVh9oECRIxWtsDlf7qQ5V+Dev21B16Ln56UhwzCzL5yNmBSn9GfoZG7nTX0hCoxI+o5Mu7VvYtPczoF5cEaaMgLQ9GzYRTL4XkrE4VcrfKPC4xWIknHX49VC6pU+UtvQlpH0HwnoBnu637aqf3a4AFoYxB5Gia29pZsX0/r22s5PXNVazZU3toyGZaUhwz8jO47ewiZuRnMDM/k8LsKK70WxsPV+L1ZUc5iy+D5poj941NDFTuaaNg5DSYcOHh5bQ8GJYXeE3K6HR2LAMp3J3F/cbdI/4fqS6WTo67s6G8nlc3VvDqxkre3LqPxtZ24mKM2YWZ3PqeokBHbn4GY7JTIvcpmq2N0FgNTTXQFHztvtxQ1fVMvqn6yOPEJhyu0IdPgvHnBZfzulb0SZmq4Ae5iEgESUlJVFVVkZOTE7HJwN2pqqoiKUnDC4/H3tomXttUyWsbK3ltU+WhWbHGD0/lurkFLJw4nDPHZw+th6x1tENzbe+V+dGW25uPcXAgYVigSSYtD3InwLiFMGzk4Yr94Gtylir4CBERiaCgoIDS0lIqKoZsP3OfJCUlUVDQ/0PHIkljSzv/3Fp1qOJfVxZoi85OTWDBhFwWTshlwcRc8jOTwxxpkDs07IOanVBTGjj7bqwOVt7VXSvzxprA++ZaDg077InFQnJmoKklKfiaUdB1ufP25Mzg+kxISofYIZQUpV9ERCKIj49n3LjQ9KbL4NbR4ZTsruWVjRW8trGSFdv309LeQUJcDGcUZfGFyyazcGIuU0elh6epp7UJancFKvlDfzu7Lrc1HrlffGrXyjq9AEZO76UyD75PSNWZuhyXiEgEEl1K9zfw2sZKXt1UyeubKtnfEBj3PWVUOrcuKOLsCbmcUZRNckKIR424w4HKIyv2zssHerglZlhe4Ax95LTAyJiMwsByRgGkjw40ueisXAaQEoEMenVNrSzdXMVrmyp5dWMlWysDY8tHpidyweSRLJyYy4IJuQxPS+zfD25thJpdx67ou7e3x6ccrtjzZnSt5A9W9HH9HKfISVIikEGnrb2Dd0qreWVDoJ1/5c5q2juclIRY5o/P4Zb5Y1k4MZcJI4ad3OCA5nrYv+3wX/dKvqGy2w4W6CjNKIBRs2DyFUdW9OpAlSFIiUDCzt3ZWnng0Bn/G5urqGtuI8ZgRkEmHzv3FM6emMtpY7KOb3KVjo5A5+uhyn5r14r/QLfBBQnDDlfso+cEK/dOFX3aKIhL6LfvLTJYKBFI2GyvOsAvXt3KS+v2sqs60GFamJ3M+2aN5pyJubznlNzeJ1lpaehauXep7Ld3bbqxmECFnlUEk94beM0qguxxkDlWZ/MStZQIZMBtrqjnvsWbeGrlbuJijPMnjeBj553Cwom5jM1J7VrYPfC4gn1be67w68u7lk9Ig+yiwA1Op17WtbLPKFQnrEgPlAhkwGwor+PHL23imVW7SYqL5V8XFHH7OeMZkeRQvQMqX4WN27pV9tu7Da+0w2f1Ey8OPIQrqyjwmj1OZ/UiJ0CJQEJuze5afrx4I8+uLiM1IZbPzR/GLXnbSdv9FPzytUAS6Cw+NVCp50yACRcdruiziiCzUKNuRPqZEoGEzOrSGn740kbeXrOB8xPX8fTY7UxrfofYt7cGCiRnQ9HZMOeWTmf2RZCaq7N6kQGkRCD97p0NW3j1+SdJ27OUu+LWckpSaWBDdTqMXQBnLoJx58CIqRBzHKOARCQklAjk5DXVwvbX2fPOczRvfJkZLVuYZU5rQhI29j1wyu2BB5flzYJY/S8nMtjoX6Ucv5YG2PkGbH0F3/oq7H4b83ayPZ5VMZOoHP9Rpp99JUljz9C4e5EhQIlAetfWDKXLYOsrsPXVwPuOVjosjg1xp/Jc65WsS5rNmQsv47qzTg39M35EpF8pEciR2lth99vBiv8V2PnPwCTfFoOPmsX2U2/loT2F/KG8gMyMTD52xSl8b24hSfFKACJDkRKBBCY6KVsdqPS3vQrbX4eW+sC2kdPh9NvoKFrIS40Tufe1ckq21FKYncxXr5nANacVHN9jH0Rk0FEiiFbVO2D93w5X/k3BuWZzJsLM6wOdu0ULaU/O4W/v7uFHf9/E+vKNjMtN5bsfmsXVs0cTH6sEIBIJlAiijTusfBj++n8Dd+xmjoUpV8K4c6FoIaSPAgJPAP3Lqj386KUlbK44wIQRw/jBDbO5YsYo4pQARCKKEkE0aa6Dv34WVv0+MI7/fd+HnFO6FGlt7+DPb+/ivsWb2FbVwOS8NH784TlcPn0UsZE6mbtIlFMiiBZ7VsHjt8G+LXD+l2HhZyHmcOduS1sHT7xVyn2LN1G6v5Fpo9P52c2nc8nUkeGZ4lFEBowSQaRzh+W/hL9/CVKy4V+eCTzWIaiptZ0/Lt/JT1/ezO6aJmYVZvKNq6ZxweQRJzfpi4gMGUoEkayxGp75BKx5CiZcDB/4WeA5PkBjSzuPvrmD+1/ZTHltM6ePzeL/XTuTcybmKgGIRJmQJgIzuwz4ARAL/MLd7+m2/V7g/OBiCjDC3TNDGVPU2LUC/ngb1O6Ci++Gsz5+6Lk+NQ2tfPBnr7Nxbz3zx2dz73WzOeuUHCUAkSgVskRgZrHAfcDFQCmwzMyedvc1B8u4+6c7lf84MCdU8UQNd3jjJ/D81yAtD277GxTOO7S5pa2Dj/5uBduqDvDgrXO5YPLIMAYrIoNBKK8I5gGb3H0LgJk9BlwNrDlK+RuBr4UwnsjXsA/+/DHY8HeY/D64+seBiVqC3J0vPbmapVuquPf6WUoCIgKENhHkAzs7LZcCZ/ZU0MzGAuOAl0IYT2TbvhSe+EhgQvbLvwPzFh3xTP/7Fm/i8RWlfPLCiXxgTkGYAhWRwWawdBbfADzu7u09bTSzRcAigDFjxgxkXINfRwf841546VuQOQY+8hyMPrKF7amVu/jucxv4wJx8PnXRxDAEKiKDVSgTwS6gsNNyQXBdT24A7jjagdz9AeABgLlz53p/BTjk1e+FPy2CLYth+rWBG8SS0o8otnzbPj73x1XMK8rmnmtnqFNYRLoIZSJYBkw0s3EEEsANwIe7FzKzyUAWsDSEsUSeLS8HkkBTDVz5Qzjt//Q4veO2ygPc/pvl5Gclc/8tp5MYpyeEikhXIUsE7t5mZncCxQSGjz7o7iVmdjew3N2fDha9AXjM3XWm3xftbbDkv+GV/4HcU+GWJ2HktB6LVje08K8PLQPgV7eeQVaqJokRkSOFtI/A3Z8Fnu227qvdlr8eyhgiSu1ueOLfYPs/YPbN8N7vQEJqj0Wb29pZ9NsVlO5v5OHbz6Qot+dyIiKDpbNYerPhOfjzR6G1CT5wP8y64ahF3Z0vPrGaN7fu4wc3zOaMouwBDFREhholgsGuvRVevBte/2FgkpgPPQS5xx7188MXN/Gnt3fxmYtP5erZ+QMTp4gMWUoEg9n+7YF7A0qXwdyPwKXfgvjkY+7y57d3ce8LG7jmtHw+fsGEAQpURIYyJYLBau0z8NQdgUdGfOghmPaBXnd5c+s+Pv/4KuaPz+aea2ZqmKiI9IkSwWDT1gzP/Se8eT+MPg0++CBkj+t1t62VB1j02+UUZCXzs5tP1zzCItJnSgSDSdXmwOQxe96Bs+6EC78Gcb0P+dx/oIXbfvUmMWb86rYzyEzRMFER6TslgsFi9ePwzKcgNg5ufAwmXd6n3QLDRJezu6aJR/7tTMbmaJioiBwfJYJwa2mAv98Fb/0aCufDB38JGX17IJy78/nHV7Fs235+eOMc5mqYqIicACWCcNq7LtAUtHdtYA7h874UuCLoo++/sJGnVu7mc5dO4qpZo0MYqIhEMiWCcHCHlQ/Ds58L3Bl88xMw4cLjOsQTK0r5wYsb+dDpBfzHeaeEKFARiQZKBAOtuR7++hlY9XsYdw5c8/PATGLH4Y0tVdz1p1WcNT6Hb31ATxMVkZOjRDDQnvx3WP8snP/lQHNQzPE9DXRzRT3//tsVjMlO0TBREekXSgQDadcKWPeXQBI49/PHvXtVfTO3/WoZcTHGr26dR0ZKfAiCFJFoo0QwkF76JqTkwPyPHfeuTa2Bp4mW1zbx6KL5jMlJCUGAIhKN1K4wULb9Aza/BGd/GhLTjmvXjg7nc4+vYsX2/XzvutmcNiar951ERPpIiWAguAeuBoblBR4ed5zufWEDz7yzm89fNokrZo4KQYAiEs2UCAbClsWw43U45/9CwvE16fxx+U5+9NImrp9byMfO1TBREel/SgSh5g4v/hdkFAbmFT4Or2+q5It/Ws2CCTl88wPTNUxUREJCiSDU1v8Ndr8VGCUUl9jn3Tbtreejv1vBuNxUfnLT6cTH6j+ViISGapdQ6uiAxd+C7PEw68N93q2qvpnbHnqThLgYHrz1DDKSNUxUREJHw0dDac2fofxduOYXfX6GUFNrO7f/Zjl7a5t5bNF8CrM1TFREQkuJIFTa22Dxt2H4FJh+TZ926ehwPvvHd3hrRzU/vek05miYqIgMACWCUFn9B6jaCNf9ts+Pkfjuc+v566o9fPHyyVw+Q8NERWRgqI8gFNpa4OV7YNQsmHJln3b5/bId/OTlzdw4bwyLzhkf4gBFRA7TFUEorPwdVG+HK/4X+jDk8x+bKvnyk++ycGIud189TcNERWRA6Yqgv7U2wZL/gcIzYcJFvRbfWF7HR3+3gvHDU7nvptM0TFREBlxIax0zu8zM1pvZJjO76yhlrjOzNWZWYmaPhDKeAbH8QajbDRd8pdergYq6Zm57aBmJcbE8eOsZpCdpmKiIDLyQNQ2ZWSxwH3AxUAosM7On3X1NpzITgS8CC9x9v5mNCFU8A6K5Hl77XmDCmXHnHLNoY0s7//ab5VTWN/P7RWdRkKVhoiISHqG8IpgHbHL3Le7eAjwGXN2tzO3Afe6+H8Dd94YwntB78wE4UAEX/Ocxi3V0OJ/5w0pWlVbz/evnMKswc2DiExHpQSgTQT6ws9NyaXBdZ6cCp5rZP8zsDTO7rKcDmdkiM1tuZssrKipCFO5JaqqBf/wAJl4KhfOOWfS/i9fxt3fL+PJ7p3DZ9OObplJEpL+Fu2cyDpgInAfcCPzczDK7F3L3B9x9rrvPHT58+MBG2FdLfwJN1XD+l45Z7E9vlXL/ki3cdOYYPnL2uIGJTUTkGEKZCHYBhZ2WC4LrOisFnnb3VnffCmwgkBiGloZ9sPQ+mHIVjJ59zKIPvLKF6fnpfOMqDRMVkcEhlIlgGTDRzMaZWQJwA/B0tzJ/JnA1gJnlEmgq2hLCmELjH9+Hlvperwa2Vx1gXVkd75+dT5yGiYrIIBGy2sjd24A7gWJgLfAHdy8xs7vN7KpgsWKgyszWAIuBz7l7VahiCom6cvjnAzDzOhgx5ZhFi0vKALh0mvoFRGTwCOmdxe7+LPBst3Vf7fTegc8E/4am174H7S1w7hd6LVpcUs6UUel6oqiIDCpqnzgZ1TsDN5DNuQlyjj2N5N66Jt7asZ9Lp40coOBERPpGieBkvPI/gddzPt9r0efXlOOuZiERGXxOKBGY2bD+DmTIqdoMb/8OTr8VMgt7LV5cUs6Y7BQm56WFPjYRkeNwolcEa3ovEuGWfAdiE2DhZ3stWtvUytLNlVw6baSGjIrIoHPUzmIzO1oHrgHRfUWwdx2s+j285+OQ1ntTz+J1e2ltdzULicigdKwrgm8DWUBat79hvewX+V7+NiSkwoJP9al4cUkZucMSOU1TT4rIIHSs4aNvAX929xXdN5jZv4UupEFuzzuw5qlAB3FqTq/Fm1rbeXl9BVfPzicmRs1CIjL4HOvMfhew3cw+2cO2uSGKZ/Bb/G1IyoSz7uhT8dc2VtLQ0q5hoyIyaB0rEUwFEoB/NbMsM8s++Ae0Dkx4g8zOZbDh77DgE5Cc2addikvKSEuM4z2n5IY2NhGRE3SspqH7gReB8cAKAp3EB3lwfXR56b8gJRfm/Xufire1d/DC2nLOnzyChLjo7lYRkcHrqLWTu//Q3acAD7r7eHcf1+kv+pLA1ldg6xJY+BlI7NugqWXb9rO/oVWjhURkUOv1NNXdPzYQgQxq7vDStyBtFMz91z7vVlxSRkJcDOdNGqRzKIiIEO3DQPtq04uw8w0453MQn9ynXdyd59eUs3BCLqmJIX22n4jISVEi6I17oG8gcwzMuaXPu727q5Zd1Y1qFhKRQU+JoDfr/gJ7VsK5d0FcQp93Ky4pI8bgwikjQhebiEg/UCI4lo72QN9AzgSYef1x7VpcUsYZRdnkDEsMUXAiIv1DieBYSp6EirVw3hchtu/t/Fsq6tm4t17NQiIyJCgRHE17W+Au4hHTYNo1x7VrcUk5AJfobmIRGQI0nOVo3nkU9m2GGx6BmOPLl8UlZUzPT6cgS1NSisjgpyuCnrQ1w5L/htFzYNJ7j2vXspomVu6s5tKpahYSkaFBVwQ9ees3ULMTrvw+HOdEMs+vKQPg0ulKBCIyNOiKoLvWRnjluzDmLDjlwuPevbiknHG5qUwcEd1z94jI0KFE0N2yX0B9GVzwn8d9NVDT0MobW6q4dFqepqQUkSFDiaCz5jp47V4Yfz4ULTju3V9cV05bh2vuAREZUpQIOvvnz6ChCi74ygntXlxSxsj0RGYVZPZvXCIiIaREcFDjfvjHj+DUy6Hg+Cdga2xpZ8mGCi6ZmqcpKUVkSAlpIjCzy8xsvZltMrO7eth+q5lVmNnK4F/45kJeeh8018AFXz6h3V/ZWEFTa4fuJhaRISdkw0fNLBa4D7gYKAWWmdnT7r6mW9Hfu/udoYqjTw5Uwhs/hWkfgLwZJ3SI4pIyMpLjOXN8dj8HJyISWqG8IpgHbHL3Le7eAjwGXB3Czztxr90LrQ2BZwqdgNb2Dl5cu5cLJ48gPlatbSIytISy1soHdnZaLg2u6+5aM1tlZo+bWWFPBzKzRWa23MyWV1RU9G+UtXsCQ0ZnXg/DJ53QId7cuo+axlYuUbOQiAxB4T59fQYocveZwPPAr3sq5O4PuPtcd587fHg/T/v46v9CRxuc+4UTPkRxSRlJ8TGce6qmpBSRoSeUiWAX0PkMvyC47hB3r3L35uDiL4DTQxjPkfZvhxUPBWYeyx53Qofo6HCeKynnnInDSU6I7d/4REQGQCgTwTJgopmNM7ME4Abg6c4FzGxUp8WrgLUhjOdIr3wHLCYwF/EJWrWrhrLaJo0WEpEhK2Sjhty9zczuBIqBWOBBdy8xs7uB5e7+NPAJM7sKaAP2AbeGKp4jVG6ClY/CvEWQ0VPXRd8Ul5QRG2OaklJEhqyQPn3U3Z8Fnu227qud3n8ROLGhOidryT0Qlwhnf/qkDlNcUsb88dlkpvR9PmMRkcEk3J3F4VG+BlY/HrgaSDvx5wJt2lvHlooDahYSkSEtOhPB4m9BYhos+ORJHebQlJSahEZEhrDoSwS734Z1f4Gz7oCUk7sLuLikjFmFmeRlJPVTcCIiAy/6EsFL34LkLJj/sZM6zO7qRlaV1uiR0yIy5EVXItjxBmx6PtAklJRxUod6riQ4JaX6B0RkiIueROAOL30TUkcEOolPUnFJORNGDOOU4ZqSUkSGtuhJBFuXwLZXYeFnISH1pA61/0ALb27bp2YhEYkI0ZMIanbB8Mlw+q0nfagX1pbT3uFqFhKRiBDSG8oGlTk3wawbIebkc19xSTmjM5KYkX9y/QwiIoNB9FwRQL8kgYaWNl7dWMEl0/Iw05SUIjL0RVci6AdL1lfQ3NbBJeofEJEIoURwnIpLyshKiWdekaakFJHIoERwHFraOnhx3V4unDKSOE1JKSIRQrXZcVi6pYq6pjaNFhKRiKJEcByKS8pISYhl4cTccIciItJvlAj6qKPDeX5NOeeeOpykeE1JKSKRQ4mgj97euZ+KumY1C4lIxFEi6KPiknLiYozzJ2tKShGJLEoEfeDuFJeUcdYpOWQkx4c7HBGRfqVE0Afry+vYXtWgZiERiUhKBH1Q/G45ZnDJVN1NLCKRR4mgD4pLyjhtTBYj0jUlpYhEHiWCXuzc18CaPbWae0BEIpYSQS+KNSWliEQ4JYJePFdSzuS8NMbmnNysZiIig5USwTFU1jezbPs+LtHVgIhEsJAmAjO7zMzWm9kmM7vrGOWuNTM3s7mhjOd4vbCmHHfUPyAiES1kicDMYoH7gMuBqcCNZja1h3JpwCeBf4YqlhNVXFJGQVYyU0elhzsUEZGQCeUVwTxgk7tvcfcW4DHg6h7K/Rfw30BTCGM5bnVNrfxjUxWXakpKEYlwoUwE+cDOTsulwXWHmNlpQKG7//VYBzKzRWa23MyWV1RU9H+kPXh5fQUt7R0aLSQiES9sncVmFgN8D/hsb2Xd/QF3n+vuc4cPHx764Ag0C+WkJnD62KwB+TwRkXAJZSLYBRR2Wi4IrjsoDZgOvGxm24D5wNODocO4ua2dl9dXcPHUkcTGqFlIRCJbKBPBMmCimY0zswTgBuDpgxvdvcbdc929yN2LgDeAq9x9eQhj6pPXN1VR36wpKUUkOoQsEbh7G3AnUAysBf7g7iVmdreZXRWqz+0PxSVlDEuM4z0TcsIdiohIyMWF8uDu/izwbLd1Xz1K2fNCGUtftQenpDxv0nAS4zQlpYhEPt1Z3M2K7fupOtCiZiERiRpKBN0Ul5SREBvDeZMGZnSSiEi4KRF0cnBKygUTckhL0pSUIhIdlAg6WbOnltL9jWoWEpGookTQSXFJOTEGF2lKShGJIkoEnTxXUsbcsdnkDksMdygiIgNGiSBoe9UB1pXVcYkeOS0iUUaJIEhTUopItFIiCCouKWfqqHQKs1PCHYqIyIBSIgD21jXx1o79uhoQkaikRAA8f3BKyunqHxCR6KNEQKBZaGxOCpNGpoU7FBGRARf1iaC2qZWlmys1JaWIRK2oTwSL1+2ltd25VMNGRSRKRX0iKC4pY3haInMKNSWliESnqE4ETa2Hp6SM0ZSUIhKlojoRvLaxkoaWdg0bFZGoFtWJoLikjLSkOM4arykpRSR6RW0iaGvv4IW15VwweQQJcVH7M4iIRG8iWLZtP/sbWtUsJCJRL2oTQXFJGYlxMZx7qqakFJHoFpWJwN15fk05CycOJzUxLtzhiIiEVVQmgnd31bKrulE3kYmIEKWJoLikjNgY46IpSgQiIlGbCOYVZZOVmhDuUEREwi7qEsGWino27q1Xs5CISFBIE4GZXWZm681sk5nd1cP2j5rZajNbaWavmdnUUMYDgUdOA1yiYaMiIkAIE4GZxQL3AZcDU4Ebe6joH3H3Ge4+G/gO8L1QxXNQcUkZMwsyGJ2ZHOqPEhEZEkJ5RTAP2OTuW9y9BXgMuLpzAXev7bSYCngI46GspomVO6t1E5mISCehHESfD+zstFwKnNm9kJndAXwGSAAu6OlAZrYIWAQwZsyYEw7ouTVlAOofEBHpJOydxe5+n7ufAnwB+MpRyjzg7nPdfe7w4Sd+J3BxSRnjh6cyYYSmpBQROSiUiWAXUNhpuSC47mgeA94fqmCqG1p4Y8s+NQuJiHQTykSwDJhoZuPMLAG4AXi6cwEzm9hp8QpgY6iCeXHtXto7XIlARKSbkPURuHubmd0JFAOxwIPuXmJmdwPL3f1p4E4zuwhoBfYD/xKqeNKT47l46khm5meE6iNERIYkcw/pQJ1+N3fuXF++fHm4wxARGVLMbIW7z+1pW9g7i0VEJLyUCEREopwSgYhIlFMiEBGJckoEIiJRTolARCTKKRGIiEQ5JQIRkSg35G4oM7MKYPsJ7p4LVPZjOEOdfo+u9Hscpt+iq0j4Pca6e49P7RxyieBkmNnyo91ZF430e3Sl3+Mw/RZdRfrvoaYhEZEop0QgIhLloi0RPBDuAAYZ/R5d6fc4TL9FVxH9e0RVH4GIiBwp2q4IRESkGyUCEZEoFzWJwMwuM7P1ZrbJzO4KdzzhYmaFZrbYzNaYWYmZfTLcMQ0GZhZrZm+b2V/CHUu4mVmmmT1uZuvMbK2ZnRXumMLFzD4d/Hfyrpk9amZJ4Y4pFKIiEZhZLHAfcDkwFbjRzKaGN6qwaQM+6+5TgfnAHVH8W3T2SWBtuIMYJH4A/N3dJwOziNLfxczygU8Ac919OoEpd28Ib1ShERWJAJgHbHL3Le7eAjwGXB3mmMLC3fe4+1vB93UE/pHnhzeq8DKzAuAK4BfhjiXczCwDOAf4JYC7t7h7dViDCq84INnM4oAUYHeY4wmJaEkE+cDOTsulRHnlB2BmRcAc4J9hDiXcvg98HugIcxyDwTigAvhVsKnsF2aWGu6gwsHddwHfBXYAe4Aad38uvFGFRrQkAunGzIYBTwCfcvfacMcTLmb2PmCvu68IdyyDRBxwGvBTd58DHACisk/NzLIItByMA0YDqWZ2c3ijCo1oSQS7gMJOywXBdVHJzOIJJIGH3f1P4Y4nzBYAV5nZNgJNhheY2e/CG1JYlQKl7n7wKvFxAokhGl0EbHX3CndvBf4EvCfMMYVEtCSCZcBEMxtnZgkEOnyeDnNMYWFmRqD9d627fy/c8YSbu3/R3QvcvYjA/xcvuXtEnvX1hbuXATvNbFJw1YXAmjCGFE47gPlmlhL8d3MhEdpxHhfuAAaCu7eZ2Z1AMYGe/wfdvSTMYYXLAuAWYLWZrQyu+5K7Pxu+kGSQ+TjwcPCkaQtwW5jjCQt3/6eZPQ68RWC03dtE6KMm9IgJEZEoFy1NQyIichRKBCIiUU6JQEQkyikRiIhEOSUCEZEop0QgMoDM7Dw94VQGGyUCEZEop0Qg0gMzu9nM3jSzlWZ2f3C+gnozuzf4fPoXzWx4sOxsM3vDzFaZ2ZPBZ9RgZhPM7AUze8fM3jKzU4KHH9bpef8PB+9aFQkbJQKRbsxsCnA9sMDdZwPtwE1AKrDc3acBS4CvBXf5DfAFd58JrO60/mHgPnefReAZNXuC6+cAnyIwN8Z4And7i4RNVDxiQuQ4XQicDiwLnqwnA3sJPKb698EyvwP+FHx+f6a7Lwmu/zXwRzNLA/Ld/UkAd28CCB7vTXcvDS6vBIqA10L+rUSOQolA5EgG/Nrdv9hlpdl/dit3os9nae70vh39O5QwU9OQyJFeBD5oZiMAzCzbzMYS+PfywWCZDwOvuXsNsN/MFgbX3wIsCc7+Vmpm7w8eI9HMUgbyS4j0lc5ERLpx9zVm9hXgOTOLAVqBOwhM0jIvuG0vgX4EgH8Bfhas6Ds/rfMW4H4zuzt4jA8N4NcQ6TM9fVSkj8ys3t2HhTsOkf6mpiERkSinKwIRkSinKwIRkSinRCAiEuWUCEREopwSgYhIlFMiEBGJcv8fkjp81Z9jbKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(f1s)\n",
    "plt.plot(f1s_eval)\n",
    "plt.title('model f1 micro')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7343342911877394"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(f1s_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Домашнее задание"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Соберите 1 сетку со всеми разобранными элементами - несколькими свертками, пулингами, батчнормами, активациями, дропаутами и регуляризацией. В вашей архитектуре должны быть как наслоенные (CNN-CNN-CNN-POOLING, например) друг на друга, так и параллельные элементы (CNN-POOLING; CNN-POOLING с конткатенацией выходов первого и второго перед flatten в 1 вектор) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Запустите обучение и напечатайте результат. Важно, чтобы получилось не хуже, чем на картинках выше (т.е. качество хотя бы 0.75)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}