examples/ChatGLM2_LoRA——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ChatGLM2-6b微调保姆级教程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "😋😋公众号算法美食屋后台回复关键词：**torchkeras**，获取本文notebook源代码和数据集下载链接。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "干货预警：这可能是你能够找到的最容易懂的，最完整的，适用于各种NLP任务的开源LLM的finetune教程~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChatGLM2-6b是清华开源的小尺寸LLM，只需要一块普通的显卡(32G较稳妥)即可推理和微调，是目前社区非常活跃的一个开源LLM。\n",
    "\n",
    "本范例使用非常简单的，外卖评论数据集来实施微调，让ChatGLM2-6b来对一段外卖评论区分是好评还是差评。\n",
    "\n",
    "可以发现，经过微调后的模型，相比直接 3-shot-prompt 可以取得明显更好的效果。\n",
    "\n",
    "值得注意的是，尽管我们以文本分类任务为例，实际上，任何NLP任务，例如，命名实体识别，翻译，聊天对话等等，都可以通过加上合适的上下文，转换成一个对话问题，并针对我们的使用场景，设计出合适的数据集来微调ChatGLM2.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#安装环境\n",
    "\n",
    "#chatglm\n",
    "!pip install transformers\n",
    "\n",
    "\n",
    "#finetune\n",
    "!pip install -U accelerate\n",
    "!pip install datasets\n",
    "!pip install -U peft \n",
    "!pip install -U torchkeras \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "公众号算法美食屋后台回复关键词： torchkeras，获取本文notebook源代码，以及waimai数据集下载链接~\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要从 https://huggingface.co/THUDM/chatglm2-6b 下载chatglm2的模型。\n",
    "\n",
    "国内可能速度会比较慢，总共有14多个G，网速不太好的话，大概可能需要一两个小时。\n",
    "\n",
    "如果网络不稳定，也可以手动从这个页面一个一个下载全部文件然后放置到 一个文件夹中例如 'chatglm2-6b' 以便读取。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-17 09:48:19,269] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9189c5971eb540a09a74ce3041515b0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import  AutoModel,AutoTokenizer\n",
    "model_name = \"chatglm2-6b\" #或者远程 “THUDM/chatglm2-6b”\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_name, trust_remote_code=True)\n",
    "model = AutoModel.from_pretrained(model_name,trust_remote_code=True).half().cuda()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\"\"文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\n",
    "\n",
    "下面是一些范例:\n",
    "\n",
    "味道真不错 -> 好评\n",
    "太辣了，吃不下都  -> 差评\n",
    "\n",
    "请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\n",
    "\n",
    "xxxxxx ->\n",
    "\"\"\"\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prompt.replace('xxxxxx',text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评\n"
     ]
    }
   ],
   "source": [
    "response, his = model.chat(tokenizer, get_prompt('味道不错，下次还来'), history=[])\n",
    "print(response)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#增加4个范例\n",
    "his.append((\"太贵了 -> \",\"差评\"))\n",
    "his.append((\"非常快，味道好 -> \",\"好评\"))\n",
    "\n",
    "his.append((\"这么咸真的是醉了 -> \",\"差评\"))\n",
    "his.append((\"价格感人 优惠多多 -> \",\"好评\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们来测试一下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "差评\n",
      "差评\n",
      "好评\n"
     ]
    }
   ],
   "source": [
    "response, history = model.chat(tokenizer, \"一言难尽啊 -> \", history=his)\n",
    "print(response) \n",
    "\n",
    "response, history = model.chat(tokenizer, \"还凑合一般般 -> \", history=his)\n",
    "print(response) \n",
    "\n",
    "response, history = model.chat(tokenizer, \"我家狗狗爱吃的 -> \", history=his)\n",
    "print(response) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'差评'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#封装成一个函数吧~\n",
    "def predict(text):\n",
    "    response, history = model.chat(tokenizer, f\"{text} ->\", history=his,\n",
    "    temperature=0.01)\n",
    "    return response \n",
    "\n",
    "predict('死鬼，咋弄得这么有滋味呢') #在我们精心设计的一个评论下，ChatGLM2-6b终于预测错误了~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们拿外卖数据集测试一下未经微调，纯粹的 6-shot prompt 的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评    4000\n",
      "差评    4000\n",
      "Name: tag, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "import datasets \n",
    "\n",
    "\n",
    "df = pd.read_csv(\"data/waimai_10k.csv\")\n",
    "\n",
    "df['tag'] = df['label'].map({0:'差评',1:'好评'})\n",
    "df = df.rename({'review':'text'},axis = 1)\n",
    "\n",
    "dfgood = df.query('tag==\"好评\"')\n",
    "dfbad = df.query('tag==\"差评\"').head(len(dfgood)) #采样部分差评，让好评差评平衡\n",
    "df = pd.concat([dfgood,dfbad])\n",
    "\n",
    "\n",
    "print(df['tag'].value_counts())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_dic = datasets.Dataset.from_pandas(df).train_test_split(\n",
    "    test_size = 2000,shuffle=True, seed = 43)\n",
    "dftrain = ds_dic['train'].to_pandas()\n",
    "dftest = ds_dic['test'].to_pandas()\n",
    "dftrain.to_parquet('data/dftrain.parquet')\n",
    "dftest.to_parquet('data/dftest.parquet')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = ['' for x in dftest['tag']] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [03:06<00:00, 10.75it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm \n",
    "for i in tqdm(range(len(dftest))):\n",
    "    text = dftest['text'].loc[i]\n",
    "    preds[i] = predict(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftest['pred'] = preds "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>pred</th>\n",
       "      <th>好评</th>\n",
       "      <th>差评</th>\n",
       "      <th>无法确定评论的具体内容,因为只看到了一次评论。无法进行分类。</th>\n",
       "      <th>无法确定这是好评还是差评,因为评论中没有提到菜品数量和味道。请提供更多信息。</th>\n",
       "      <th>负面</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tag</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>好评</th>\n",
       "      <td>809.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>差评</th>\n",
       "      <td>56.0</td>\n",
       "      <td>947.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "pred     好评     差评  无法确定评论的具体内容,因为只看到了一次评论。无法进行分类。  \\\n",
       "tag                                                  \n",
       "好评    809.0  185.0                             NaN   \n",
       "差评     56.0  947.0                             1.0   \n",
       "\n",
       "pred  无法确定这是好评还是差评,因为评论中没有提到菜品数量和味道。请提供更多信息。   负面  \n",
       "tag                                                \n",
       "好评                                       NaN  NaN  \n",
       "差评                                       1.0  1.0  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.pivot_table(index='tag',columns = 'pred',values='text',aggfunc='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = len(dftest.query('tag==pred'))/len(dftest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc= 0.878\n"
     ]
    }
   ],
   "source": [
    "print('acc=',acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，微调之前，我们的模型准确率为87.8%，下面我们通过6000条左右数据的微调，看看能否把acc打上去~ 😋 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要把数据整理成对话的形式，即 context 和 target 的配对，然后拼到一起作为一条样本。\n",
    "\n",
    "ChatGLM模型本质上做的是一个文字接龙的游戏，即给定一段话的上半部分，它会去续写下半部分。\n",
    "\n",
    "我们这里指定上半部分为我们设计的文本分类任务的prompt，下半部分为文本分类结果。\n",
    "\n",
    "所以我们微调的目标就是让它预测的下半部分跟我们的设定的文本分类一致。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1，数据加载"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np  \n",
    "import datasets \n",
    "\n",
    "dftrain = pd.read_parquet('data/dftrain.parquet')\n",
    "dftest = pd.read_parquet('data/dftest.parquet')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "好评    3006\n",
       "差评    2994\n",
       "Name: tag, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain['tag'].value_counts() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#将上下文整理成与推理时候一致，参照model.chat中的源码~\n",
    "#model.build_inputs??\n",
    "def build_inputs(query, history):\n",
    "    prompt = \"\"\n",
    "    for i, (old_query, response) in enumerate(history):\n",
    "        prompt += \"[Round {}]\\n\\n问：{}\\n\\n答：{}\\n\\n\".format(i + 1, old_query, response)\n",
    "    prompt += \"[Round {}]\\n\\n问：{} -> \\n\\n答：\".format(len(history) + 1, query)\n",
    "    return prompt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Round 1]\n",
      "\n",
      "问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\n",
      "\n",
      "下面是一些范例:\n",
      "\n",
      "味道真不错 -> 好评\n",
      "太辣了，吃不下都  -> 差评\n",
      "\n",
      "请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\n",
      "\n",
      "味道不错，下次还来 ->\n",
      "\n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 2]\n",
      "\n",
      "问：太贵了 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 3]\n",
      "\n",
      "问：非常快，味道好 -> \n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 4]\n",
      "\n",
      "问：这么咸真的是醉了 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 5]\n",
      "\n",
      "问：价格感人 优惠多多 -> \n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 6]\n",
      "\n",
      "问：味道不太行 -> \n",
      "\n",
      "答：\n"
     ]
    }
   ],
   "source": [
    "print(build_inputs('味道不太行',history=his))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                context target\n",
       "0     [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "1     [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "2     [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "3     [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "4     [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "...                                                 ...    ...\n",
       "1995  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "1996  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "1997  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "1998  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "1999  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "\n",
       "[2000 rows x 2 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain['context'] = [build_inputs(x,history=his) for x in dftrain['text']]\n",
    "dftrain['target'] = [x for x in dftrain['tag']]\n",
    "dftrain = dftrain[['context','target']]\n",
    "\n",
    "dftest['context'] = [build_inputs(x,history=his) for x in dftest['text']]\n",
    "dftest['target'] = [x for x in dftest['tag']]\n",
    "dftest = dftest[['context','target']]\n",
    "\n",
    "dftest "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = datasets.Dataset.from_pandas(dftrain)\n",
    "ds_val = datasets.Dataset.from_pandas(dftest)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2，token编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了将文本数据喂入模型，需要将词转换为token。\n",
    "\n",
    "也就是把context转化成context_ids，把target转化成target_ids. \n",
    "\n",
    "同时，我们还需要将context_ids和target_ids拼接到一起作为模型的input_ids。\n",
    "\n",
    "这是为什么呢？\n",
    "\n",
    "因为ChatGLM2基座模型是一个TransformerDecoder结构，是一个被预选练过的纯粹的语言模型(LLM，Large Lauguage Model)。\n",
    "\n",
    "一个纯粹的语言模型，本质上只能做一件事情，那就是计算任意一段话像'人话'的概率。\n",
    "\n",
    "我们将context和target拼接到一起作为input_ids， ChatGLM2 就可以判断这段对话像'人类对话'的概率。\n",
    "\n",
    "在训练的时候我们使用梯度下降的方法来让ChatGLM2的判断更加准确。\n",
    "\n",
    "训练完成之后，在预测的时候，我们就可以利用贪心搜索或者束搜索的方法按照最像\"人类对话\"的方式进行更合理的文本生成。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "import transformers\n",
    "\n",
    "model_name = \"chatglm2-6b\"\n",
    "max_seq_length = 512\n",
    "skip_over_length = True\n",
    "\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
    "    model_name, trust_remote_code=True)\n",
    "\n",
    "config = transformers.AutoConfig.from_pretrained(\n",
    "    model_name, trust_remote_code=True, device_map='auto')\n",
    "\n",
    "def preprocess(example):\n",
    "    context = example[\"context\"]\n",
    "    target = example[\"target\"]\n",
    "    \n",
    "    context_ids = tokenizer.encode(\n",
    "            context, \n",
    "            max_length=max_seq_length,\n",
    "            truncation=True)\n",
    "    \n",
    "    target_ids = tokenizer.encode(\n",
    "        target,\n",
    "        max_length=max_seq_length,\n",
    "        truncation=True,\n",
    "        add_special_tokens=False)\n",
    "    \n",
    "    input_ids = context_ids + target_ids + [config.eos_token_id]\n",
    "    \n",
    "    # -100标志位后面会在计算loss时会被忽略不贡献损失，我们集中优化target部分生成的loss\n",
    "    labels = [-100]*len(context_ids)+ target_ids + [config.eos_token_id]\n",
    "    \n",
    "    return {\"input_ids\": input_ids,\n",
    "            \"labels\": labels,\n",
    "            \"context_len\": len(context_ids),\n",
    "            'target_len':len(target_ids)+1}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/6000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Filter:   0%|          | 0/6000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_train_token = ds_train.map(preprocess).select_columns(['input_ids','labels', 'context_len','target_len'])\n",
    "if skip_over_length:\n",
    "    ds_train_token = ds_train_token.filter(\n",
    "        lambda example: example[\"context_len\"]<max_seq_length and example[\"target_len\"]<max_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Filter:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_val_token = ds_val.map(preprocess).select_columns(['input_ids', 'labels','context_len','target_len'])\n",
    "if skip_over_length:\n",
    "    ds_val_token = ds_val_token.filter(\n",
    "        lambda example: example[\"context_len\"]<max_seq_length and example[\"target_len\"]<max_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3, 管道构建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_collator(examples: list):\n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train_token,num_workers=2,batch_size=4,\n",
    "                                       pin_memory=True,shuffle=True,\n",
    "                                       collate_fn = data_collator)\n",
    "dl_val = torch.utils.data.DataLoader(ds_val_token,num_workers=2,batch_size=4,\n",
    "                                    pin_memory=True,shuffle=True,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[64790, 64792,   790,  ..., 55342, 55078,     2],\n",
       "         [64790, 64792,   790,  ...,     0,     0,     0],\n",
       "         [64790, 64792,   790,  ...,     0,     0,     0],\n",
       "         [64790, 64792,   790,  ...,     0,     0,     0]]),\n",
       " 'labels': tensor([[ -100,  -100,  -100,  ..., 55342, 55078,     2],\n",
       "         [ -100,  -100,  -100,  ...,  -100,  -100,  -100],\n",
       "         [ -100,  -100,  -100,  ...,  -100,  -100,  -100],\n",
       "         [ -100,  -100,  -100,  ...,  -100,  -100,  -100]])}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train.size = 300 #用约300个step做一次验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "id": "MJ2yja61CAZ_",
    "outputId": "42292006-5f8b-40d4-d60c-55704ca1aeac"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fcb52cde6a04c04bb6bc0234eb8c931",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 1,949,696 || all params: 6,245,533,696 || trainable%: 0.031217444255383614\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, AutoModel, TrainingArguments, AutoConfig\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from peft import get_peft_model, LoraConfig, TaskType\n",
    "\n",
    "model = AutoModel.from_pretrained(\"chatglm2-6b\",\n",
    "                                  load_in_8bit=False, \n",
    "                                  trust_remote_code=True)\n",
    "\n",
    "model.supports_gradient_checkpointing = True  #节约cuda\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n",
    "\n",
    "\n",
    "peft_config = LoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=8,\n",
    "    lora_alpha=32, lora_dropout=0.1,\n",
    ")\n",
    "\n",
    "model = get_peft_model(model, peft_config)\n",
    "model.is_parallelizable = True\n",
    "model.model_parallel = True\n",
    "model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_model, LoraConfig, TaskType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用我们的梦中情炉torchkeras来实现最优雅的训练循环~\n",
    "\n",
    "注意这里，为了更加高效地保存和加载参数，我们覆盖了KerasModel中的load_ckpt和save_ckpt方法，\n",
    "\n",
    "仅仅保存和加载lora权重，这样可以避免加载和保存全部模型权重造成的存储问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net(input_ids=batch[\"input_ids\"],labels=batch[\"labels\"]).loss\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "\n",
    "#仅仅保存lora可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_model = KerasModel(model,loss_fn = None,\n",
    "        optimizer=torch.optim.AdamW(model.parameters(),lr=2e-6))\n",
    "ckpt_path = 'waimai_chatglm4'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB+NklEQVR4nO3dd1xT1/sH8E8IEPbeiIK4EXFTtK6KWrXurXW2tUOtiFq136q1/irWVff8to5vnXXUtra2ilAt4kSsIlJRVESWyF6B5Pz+uCYSSSCBhJvI83698iK5Offk3FyS++Tc55wrYIwxEEIIIYTUMSO+G0AIIYSQ+omCEEIIIYTwgoIQQgghhPCCghBCCCGE8IKCEEIIIYTwgoIQQgghhPCCghBCCCGE8IKCEEIIIYTwgoIQQgghhPCCghCic19++SUEAgGePXvGd1PqzMOHDyEQCLBnzx6drkMMz6pVq9CiRQtIpVK+m1Jn9PE7YOHChQgMDOS7GfUeBSHktbVixQr89NNPfDeDvCInJwfTp0+Hs7MzLC0t0atXL8TExKi9fnx8PN5++21YWVnBwcEBEydORGZmZqVyqampmD59Onx8fGBubg5fX1+EhoYiKytLXkYqlWLPnj0YPHgwvLy8YGlpidatW+P//u//UFJSovT109PT8eGHH8LT0xNmZmbw9vbGe++9p1bb8/Ly8M0332DBggUwMqr+67ekpARlZWVq1f06S0hIwJw5c9ClSxeYmZlBIBDg4cOHSsuWlJQgLCwMrVq1goWFBTw9PTFq1CjExcUplAsJCcHNmzfx888/18EWEFWM+W4AIbqyYsUKjBw5EkOHDuW7KeQFqVSKgQMH4ubNm5g/fz6cnJywdetW9OzZE9evX0fTpk2rXP/Jkyfo3r07bG1tsWLFChQUFGDNmjW4desWrly5AlNTUwBAQUEBgoKCUFhYiE8++QReXl64efMmNm/ejIiICFy/fh1GRkYoKirC1KlT8cYbb+Cjjz6Ci4sLoqOjsXTpUoSHh+PcuXMQCATy109OTkbXrl0BAB999BE8PT3x9OlTXLlyRa3t//7771FeXo5x48apLHP16lVs3LgRf/75JzIyMiAQCODp6Ylhw4bh008/RZMmTdR6rddJdHQ0Nm7ciFatWqFly5aIjY1VWXbChAn4+eef8cEHH6B9+/Z4+vQptmzZgqCgINy6dQuNGjUCALi5uWHIkCFYs2YNBg8eXEdbQiphhOjY0qVLGQCWmZlZp69raWnJJk+eXKevKZOUlMQAsN27d+t0HUNz+PBhBoD9+OOP8mUZGRnMzs6OjRs3rtr1P/74Y2Zubs4ePXokX3bmzBkGgO3YsUO+bP/+/QwA+/XXXxXWX7JkCQPAYmJiGGOMlZaWsqioqEqvs2zZMgaAnTlzRmF5//79mY+PD3v27Jl6G/yKNm3asHfffVfpc2VlZWzGjBlMIBCwbt26sTVr1rBffvmFHTt2jK1YsYK1bduWmZmZsc2bN9fotflU2++ArKwslpeXxxhjbPXq1QwAS0pKqlTuyZMnDACbN2+ewvJz584xAGzdunUKy48ePcoEAgG7f/9+jdpFao9Ox5A68+zZM4wePRo2NjZwdHTE7NmzlXZ5//DDD+jQoQPMzc3h4OCAsWPHIjk5WaHMvXv3MGLECLi5ucHMzAwNGjTA2LFjkZubCwAQCAQoLCzE3r17IRAIIBAIMGXKFKXtSk9Ph7GxMZYtW1bpuYSEBAgEAmzevBkA8Pz5c8ybNw/+/v6wsrKCjY0N+vfvj5s3b9by3VHt3Llz6NatGywtLWFnZ4chQ4YgPj5eoUx+fj5CQkLg7e0NkUgEFxcX9OnTR+E0R3XvWV04evQoXF1dMXz4cPkyZ2dnjB49GidPnkRpaWmV6x87dgzvvPMOGjZsKF8WHByMZs2a4ciRI/JleXl5AABXV1eF9d3d3QEA5ubmAABTU1N06dKl0usMGzYMABTe57t37+L333/H/Pnz4ejoqPGpkqSkJPzzzz8IDg5W+vzUqVNx4MAB/Pbbbzh//jzmzp2Ld955B8OHD8eiRYtw48YNbN++HfPmzcP27dsrrZ+SkoJp06bB1dUVIpEIfn5++P777xXKREZGQiAQ4PDhw/j888/h5uYGS0tLDB48uNJnDAB+/PFH+WfRyckJ7777LlJSUiqVu3v3LkaPHg1nZ2eYm5ujefPm+M9//lOpXE5ODqZMmQI7OzvY2tpi6tSpKCoqqva9c3BwgLW1dbXl8vPzAVS/32Vk++LkyZPV1k10g07HkDozevRoeHt7IywsDJcuXcLGjRuRnZ2Nffv2yct8/fXXWLx4MUaPHo33338fmZmZ2LRpE7p3744bN27Azs4OYrEY/fr1Q2lpKWbNmgU3NzekpKTg119/RU5ODmxtbfG///0P77//Pjp37ozp06cDAHx9fZW2y9XVFT169MCRI0ewdOlShecOHz4MoVCIUaNGAQAePHiAn376CaNGjYKPjw/S09OxY8cO9OjRA3fu3IGHh4dW37OzZ8+if//+aNy4Mb788ksUFxdj06ZN6Nq1K2JiYuDt7Q2AOzVw9OhRzJw5E61atUJWVhb+/vtvxMfHo3379mq9Z6oUFRWpdaAQCoWwt7evssyNGzfQvn37SvkQnTt3xs6dO/Hvv//C399f6bopKSnIyMhAx44dKz3XuXNn/Pbbb/LH3bt3h5GREWbPno21a9eiQYMG+Oeff/D1119j6NChaNGiRZXtTEtLAwA4OTnJl509exYA9//Su3dvnDt3DkKhEH369MG2bdvk+0KVixcvAgDat29f6bn//e9/OHHiBC5fvgw/Pz8AAGMMhYWFsLKyAsAF8RMnToSTkxNGjRqF/v37y08tpKen44033oBAIMDMmTPh7OyM33//He+99x7y8vIQEhKi8Hpff/01BAIBFixYgIyMDKxfvx7BwcGIjY2VH6j37NmDqVOnolOnTggLC0N6ejo2bNiAqKgo+WcRAP755x9069YNJiYmmD59Ory9vXH//n388ssv+PrrrxVed/To0fDx8UFYWBhiYmLw3//+Fy4uLvjmm2+qfO/U5evriwYNGmDt2rVo3rw52rVrh6dPn+Kzzz6Dj48Pxo4dq1De1tYWvr6+iIqKwpw5c7TSBqIhvrtiyOtP1hU7ePBgheWffPIJA8Bu3rzJGGPs4cOHTCgUsq+//lqh3K1bt5ixsbF8+Y0bNyp16SujyemYHTt2MADs1q1bCstbtWrF3nrrLfnjkpISJpFIFMokJSUxkUjEvvrqK4Vl0MLpmLZt2zIXFxeWlZUlX3bz5k1mZGTEJk2aJF9ma2vLZsyYobJudd8zZWT7r7pbo0aNqq3L0tKSTZs2rdLyU6dOMQDs9OnTKte9evUqA8D27dtX6bn58+czAKykpES+7L///S+zs7NTaOPkyZNZWVlZte0MDg5mNjY2LDs7W77s008/ZQCYo6Mje/vtt9nhw4fZ6tWrmZWVFfP19WWFhYVV1vnFF18wACw/P19huVQqZT4+Pmz9+vXyZSdPnmQeHh4MAGvYsCH7448/FE5BDBs2jH3++efy8u+99x5zd3evdJpo7NixzNbWlhUVFTHGGIuIiGAAmKenp/z0BmOMHTlyhAFgGzZsYIwxJhaLmYuLC2vdujUrLi6Wl/v1118ZALZkyRL5su7duzNra2uFU2Sy7ZKR/Q+9uu+HDRvGHB0dq3zfXlXV6RjGGLt8+TLz9fVV2O8dOnRgqampSsv37duXtWzZUqM2EO2h0zGkzsyYMUPh8axZswBA/gv2+PHjkEqlGD16NJ49eya/ubm5oWnTpoiIiAAA+a/2P/74Q61f6OoYPnw4jI2NcfjwYfmy27dv486dOxgzZox8mUgkkv+Kl0gkyMrKgpWVFZo3b67RCA91pKamIjY2FlOmTIGDg4N8eZs2bdCnTx+FX/52dna4fPkynj59qrSu2rxnkyZNwpkzZ6q97d+/v9q6iouLIRKJKi03MzOTP1/VugDUXt/T0xOdO3fG+vXrceLECYSGhmL//v1YuHBhlW1csWIFzp49i5UrV8p/7QNcsivAJTSeOnUKo0ePxrx587Br1y7cv38fBw4cqLLerKwsGBsby3s2ZK5fv46MjAz5CJuUlBSMGzcOnTt3xrFjxzBnzhxMmzZNYZ2hQ4ciMjISANdjcuzYMQwaNAiMMYXPTr9+/ZCbm1vpf3PSpEkKpzdGjhwJd3d3+f/UtWvXkJGRgU8++UT+3gLAwIED0aJFC5w6dQoAkJmZifPnz2PatGkKp8gAKCT0ynz00UcKj7t164asrCz56TNtsLe3R9u2bbFw4UL89NNPWLNmDR4+fIhRo0YpPf1rb2+vV0OH6xs6HUPqzKsjH3x9fWFkZCQfanfv3j0wxlSOkDAxMQEA+Pj4IDQ0FOvWrcP+/fvRrVs3DB48GO+++26VpxWq4uTkhN69e+PIkSNYvnw5AO5UjLGxsUL+glQqxYYNG7B161YkJSVBIpHIn3N0dKzRa6vy6NEjAEDz5s0rPdeyZUv88ccfKCwshKWlJVatWoXJkyfDy8sLHTp0wIABAzBp0iQ0btwYQO3es8aNG8vrqS1zc3OleR+yg8Or5+xfXReAWutHRUXhnXfewaVLl+Snb4YOHQobGxssW7YM06ZNQ6tWrSrVc/jwYXzxxRd477338PHHHyt9/dGjRyucTho1ahQmTpyIixcv4v3331e98Spcv34dHTt2lAcn+/fvh6enJ44ePQqhUAiACzKnTp0qX8fV1VU+LDkzMxM5OTnYuXMndu7cqfQ1MjIyFB6/+hkTCARo0qSJ/LNY1f9eixYt8PfffwPgTk8CQOvWrdXa1lcDFdnpu+zsbNjY2KhVR1Vyc3PRrVs3zJ8/H3PnzpUv79ixI3r27Indu3dX2q+MMaUBE6kb1BNCePPqB18qlUIgEOD06dNKf2nv2LFDXnbt2rX4559/8Pnnn6O4uBiffvop/Pz88OTJkxq3Z+zYsfj333/lw/+OHDmC3r17K+QFrFixAqGhoejevTt++OEH/PHHHzhz5gz8/Px4nXxq9OjRePDgATZt2gQPDw+sXr0afn5++P333+VlavqeFRQUIC0trdqbsrk6XuXu7o7U1NRKy2XLqsqpkSUXqlrfwcFB3kuyY8cOuLq6VsofGTx4MBhj8vyMis6cOYNJkyZh4MCBShM/ZW17NelRKBTC0dER2dnZKtsOcEFqeXm5PHlSJisrS2G7Hz58iHbt2skDEIDLeakoOTlZHvTK/u/effddlb1UsmHFfKu4TRUxxrRS/7Fjx5Cenl5pyG2PHj1gY2ODqKioSutkZ2crfMZJ3aKeEFJn7t27Bx8fH/njxMRESKVSeUKfr68vGGPw8fFBs2bNqq3P398f/v7++OKLL3Dx4kV07doV27dvx//93/8BUN4dXJWhQ4fiww8/lJ+S+ffff7Fo0SKFMkePHkWvXr3w3XffKSzPycnR+heZLOkwISGh0nN3796Fk5MTLC0t5cvc3d3xySef4JNPPkFGRgbat2+Pr7/+Gv3795eXqe49U2bNmjVKRw4pa6+qCaRk2rZtiwsXLkAqlSr0Jly+fBkWFhZV7ndPT084Ozvj2rVrlZ67cuUK2rZtK3+cnp6u0EslIxvNUl5errD88uXLGDZsGDp27IgjR47A2LjyV2OHDh0AoNLoELFYjGfPnsHZ2Vll2wHIk2GTkpLQpk0b+XIbGxuFEUpubm6V5h2R9TgA3AH7u+++k4/scHZ2hrW1NSQSicqRN6+6d++ewmPGGBITE+Xtqvi/99ZbbymUTUhIkD8v6yG7ffu2Wq+ra+np6QBQad8zxiCRSCrtd4DbHwEBAXXSPlIZ9YSQOrNlyxaFx5s2bQIA+UFy+PDhEAqFWLZsWaVfRowx+UyXeXl5lb5M/P39YWRkpNBVb2lpiZycHLXbZ2dnh379+uHIkSM4dOgQTE1NK010JhQKK7Xtxx9/VDpssbbc3d3Rtm1b7N27V2E7bt++jT///BMDBgwAwH3hvjrM1sXFBR4eHvL3Q933TBlt5oSMHDkS6enpOH78uHzZs2fP8OOPP2LQoEEK+R7379/H/fv3FdYfMWIEfv31V4XhpOHh4fj333/lI5gAoFmzZkhPT5fnTcgcPHgQANCuXTv5svj4eAwcOBDe3t749ddfVZ4S6tmzJ1xcXLB//36F3II9e/ZAIpGgT58+VW57UFAQAFQKolq2bImrV6/KezSGDBmCGzduYMmSJXjw4AEuXLiA+fPnA+BGF40YMQJPnjzB7NmzAXD/kyNGjMCxY8eUBgPKeqj27dun0CNz9OhRpKamyj+LHTt2hIuLC7Zv367w//H777/L3y+AC4C6d++O77//Ho8fP1Z4DW31bmhCFsQeOnRIYfnPP/+MwsJChf0OcKdv7t+/r3SYNqkjfGTDkvpFlhnv7+/PBg0axLZs2cLeffddBoCNHz9eoWxYWBgDwLp06cJWrVrFtm3bxj777DPWtGlTtnr1asYYYydOnGCenp4sJCSEbd26lW3cuJF16tSJmZiYsOjoaHldAwYMYJaWlmzt2rXs4MGD7NKlS9W29YcffmAAmLW1NRs0aFCl52WTXU2ZMoXt3LmTzZo1izk4OLDGjRuzHj16yMtpa3TMmTNnmLGxMWvRogVbvXo1++qrr5izszOzt7dnDx48YIwxlp2dLR8JtG7dOrZz5042evRoBoCtXbtWo/dM18rLy9kbb7zBrKys2LJly9iWLVuYn58fs7a2Znfv3lUo26hRo0ojbh4/fswcHR2Zr68v27hxI1uxYgWzt7dn/v7+CiNj7t69yywtLZmVlRVbtGgR2759Oxs3bhwDwPr06SMvl5eXx7y8vJiRkRFbuXIl+9///qdwu3jxosLr7927lwFgnTp1Yhs3bmTz5s1jJiYmrFu3bqy8vLza7W/dunWlSdlKSkqYra0tO3HihHzZihUrmJGREQPAjI2N2YYNG+QjPfr27Svf9zJpaWmsUaNGzMLCgs2ePZvt2LGDhYWFsVGjRjF7e3t5OdnoGH9/f9amTRv27bffsoULFzIzMzPWpEkThRE+u3fvZgBYYGAgW79+PVu0aBGzsLBg3t7eCqOGYmNjmZWVFXN0dGSLFi1iO3fuZJ9//jkLCAiQl1E1WZnsNVSNdJHJyclhy5cvZ8uXL2dvv/02A8Dmzp3Lli9fzjZt2iQvV1payvz8/JhAIGBTpkxh27dvZ/PmzWNmZmbM3d290usfPXqUAWCJiYlVvj7RHQpCiM7JvoDu3LnDRo4cyaytrZm9vT2bOXOmwvA/mWPHjrE333yTWVpaMktLS9aiRQs2Y8YMlpCQwBhj7MGDB2zatGnM19eXmZmZMQcHB9arVy929uxZhXru3r3LunfvzszNzeXDM6uTl5cnL//DDz9Uer6kpITNnTuXubu7M3Nzc9a1a1cWHR3NevTooZMghDHGzp49y7p27crMzc2ZjY0NGzRoELtz5478+dLSUjZ//nwWEBDArK2tmaWlJQsICGBbt26Vl1H3PasLz58/Z++99x5zdHRkFhYWrEePHuzq1auVyikLQhhj7Pbt26xv377MwsKC2dnZsQkTJrC0tLRK5e7evctGjhzJvLy8mImJCWvUqBGbN2+ewoFW9p6ruin7nzl48CALCAhgIpGIubq6spkzZyoMd63KunXrmJWVlXzIrMzSpUtZ48aN2fPnz+XLUlJS2Pnz5+Xb9vfff7OMjAyVdaenp7MZM2bIt9fNzY317t2b7dy5U15GFoQcPHiQLVq0iLm4uDBzc3M2cODASkNsGeNmuG3Xrh0TiUTMwcGBTZgwgT158qRSudu3b7Nhw4YxOzs7ZmZmxpo3b84WL16ssH21CUKq2k+v/o88f/6czZkzhzVr1oyJRCLm5OTExo4dWylwY4yxMWPGsDfffLPK1ya6JWCMhz4zQgiph3Jzc9G4cWOsWrVK4aJ3JSUl6Nq1K4RCIU6ePClPwn3V0aNHMWzYMJUJntWJjIxEr1698OOPP2LkyJE1quN1kZaWBh8fHxw6dAhDhgzhuzn1FuWEEEJIHbG1tcVnn32G1atXK4ymMjMzw2+//QaBQIDmzZtjwYIFOH/+PB49eoS7d+9i3759CAoKwuTJk7U+H019tX79evj7+1MAwjPqCSFEx8RiMZ4/f15lGVtb2yrnyCD1g1gsxubNm7F582YkJSXJl5uZmWHYsGFYtmxZtVcargr1hBB9Q0N0CdGxixcvolevXlWW2b17t8oL7JH6w9TUFKGhoQgNDcXDhw+RkpICMzMztGzZEhYWFnw3jxCto54QQnQsOzsb169fr7KMn5+fyjwAQgh5XVEQQgghhBBeUGIqIYQQQnhBOSFKSKVSPH36FNbW1nRhI0IIIUQDjDHk5+fDw8ND4fIMylAQosTTp0/h5eXFdzMIIYQQg5WcnIwGDRpUWYaCECWsra0BcG+gNi4vTQghhNQXeXl58PLykh9Lq0JBiBKyUzA2NjYUhBBCCCE1oE46AyWmEkIIIYQXFIQQQgghhBcUhBBCCCGEF5QTQgghpM4wxlBeXg6JRMJ3U0gNCYVCGBsba2UKCwpCCCGE1AmxWIzU1FQUFRXx3RRSSxYWFnB3d4epqWmt6qEghBBCiM5JpVIkJSVBKBTCw8MDpqamNBmkAWKMQSwWIzMzE0lJSWjatGm1E5JVhYKQOiCRABcuAKmpgLs70K0bIBTy3SpCCKk7YrEYUqkUXl5edEVgA2dubg4TExM8evQIYrEYZmZmNa6LghAdO34cmD0bePLk5bIGDYANG4Dhw/lrFyGE8KE2v5qJ/tDWfqT/Bh06fhwYOVIxAAGAlBRu+fHj/LSLEEII0QcUhOiIRML1gDBW+TnZspAQrhwhhBBSH1EQoiMXLlTuAamIMSA5mStHCCFEfRIJEBkJHDzI/TWkH3Pe3t5Yv369VuqKjIyEQCBATk6OVurjA+WE6EhqqnbLEUII4SfPrmfPnmjbtq1WgoerV6/C0tKy9o16TVBPiI64u2u3HCGE1Hf6mmcnm4BNHc7OzjQ6qAIKQnSkWzcuOlc1DF4gALy8uHKEEFKfFRaqvpWUcGXUybObPVvx1Iyy+jQ1ZcoU/PXXX9iwYQMEAgEEAgH27NkDgUCA33//HR06dIBIJMLff/+N+/fvY8iQIXB1dYWVlRU6deqEs2fPKtT36ukYgUCA//73vxg2bBgsLCzQtGlT/Pzzz5o39IVjx47Bz88PIpEI3t7eWLt2rcLzW7duRdOmTWFmZgZXV1eMHDlS/tzRo0fh7+8Pc3NzODo6Ijg4GIU1edM0QEGIjgiFXPcgoDoQWb+e5gshhBArK9W3ESO4Murk2T15ophn5+1duT5NbdiwAUFBQfjggw+QmpqK1NRUeHl5AQAWLlyIlStXIj4+Hm3atEFBQQEGDBiA8PBw3LhxA2+//TYGDRqEx48fV/kay5Ytw+jRo/HPP/9gwIABmDBhAp4/f65xW69fv47Ro0dj7NixuHXrFr788kssXrwYe/bsAQBcu3YNn376Kb766iskJCTg9OnT6N69OwAgNTUV48aNw7Rp0xAfH4/IyEgMHz4cTFnUp02MVJKbm8sAsNzc3FrXdewYYw0aMMZ9RLibrS23nBBC6ovi4mJ2584dVlxcXOm5it+Pr94GDODKHDhQdTnZ7cCBl/U6OVV+viZ69OjBZs+eLX8cERHBALCffvqp2nX9/PzYpk2b5I8bNWrEvv322wrbDvbFF1/IHxcUFDAA7Pfff6+2blk7srOzGWOMjR8/nvXp00ehzPz581mrVq0YY4wdO3aM2djYsLy8vEp1Xb9+nQFgDx8+rPZ1Gat6f2pyDKWeEB0bPhx4+BCIiAAmTeKWBQXRRGWEECJTUKD6duwYV6YmeXYPH1auT5s6duyo8LigoADz5s1Dy5YtYWdnBysrK8THx1fbE9KmTRv5fUtLS9jY2CAjI0Pj9sTHx6Nr164Ky7p27Yp79+5BIpGgT58+aNSoERo3boyJEydi//798uv4BAQEoHfv3vD398eoUaOwa9cuZGdna9wGTVEQUgeEQqBnT+58JQBcvGhYQ8oIIUSXLC1V32Qzgtckz05Zfdptt2KF8+bNw4kTJ7BixQpcuHABsbGx8Pf3h1gsrrIeExMThccCgQBSqVS7jQVgbW2NmJgYHDx4EO7u7liyZAkCAgKQk5MDoVCIM2fO4Pfff0erVq2wadMmNG/eHElJSVpvR0UUhNShgABg/37g1i3KBSGEEE1UlWcne6yrPDtTU1NI1PjlGBUVhSlTpmDYsGHw9/eHm5sbHj58qP0GqdCyZUtERUVValOzZs0gfPHGGBsbIzg4GKtWrcI///yDhw8f4ty5cwC44Kdr165YtmwZbty4AVNTU5w4cUKnbaZ5QuqQUAiMH893KwghxDANHw4cPap8npD163V3mtvb2xuXL1/Gw4cPYWVlpbKXomnTpjh+/DgGDRoEgUCAxYsX66RHQ5W5c+eiU6dOWL58OcaMGYPo6Ghs3rwZW7duBQD8+uuvePDgAbp37w57e3v89ttvkEqlaN68OS5fvozw8HD07dsXLi4uuHz5MjIzM9GyZUudtpl6QgghhBiMinl2Bw5wf5OSdJtnN2/ePAiFQrRq1QrOzs4qczzWrVsHe3t7dOnSBYMGDUK/fv3Qvn173TXsFe3bt8eRI0dw6NAhtG7dGkuWLMFXX32FKVOmAADs7Oxw/PhxvPXWW2jZsiW2b9+OgwcPws/PDzY2Njh//jwGDBiAZs2a4YsvvsDatWvRv39/nbZZwJiux98Ynry8PNja2iI3Nxc2NjZarTs3F9i5E7hzB9i9W6tVE0KI3iopKUFSUhJ8fHxqdel3oh+q2p+aHEOpJ6SOGRsDixYBe/YA1SRME0IIIa81CkLqmKUlIOud+/tvfttCCCFEf3300UewsrJSevvoo4/4bp5WUGIqD958E7h6lQtCKFGVEEKIMl999RXmzZun9DltpwrwRS96QrZs2QJvb2+YmZkhMDAQV65cUVl2165d6NatG+zt7WFvb4/g4GCF8mVlZViwYAH8/f1haWkJDw8PTJo0CU+fPq2LTVHLm29yf6knhBBCiCouLi5o0qSJ0puLiwvfzdMK3oOQw4cPIzQ0FEuXLkVMTAwCAgLQr18/lbPFRUZGYty4cYiIiEB0dDS8vLzQt29fpKSkAACKiooQExODxYsXIyYmBsePH0dCQgIGDx5cl5tVJdmEdrdvA3UwIR0hhBCil3gfHRMYGIhOnTph8+bNAACpVAovLy/MmjULCxcurHZ9iUQCe3t7bN68GZNk86K/4urVq+jcuTMePXqEhg0bVlunLkfHyDRrBty7B5w6BQwYoJOXIIQQvUGjY14vr8XoGLFYjOvXryM4OFi+zMjICMHBwYiOjlarjqKiIpSVlcHBwUFlmdzcXAgEAtjZ2Sl9vrS0FHl5eQo3XXvzTUAkohEyhBBC6i9eg5Bnz55BIpHA1dVVYbmrqyvS0tLUqmPBggXw8PBQCGQqKikpwYIFCzBu3DiVEVlYWBhsbW3lN9llmnVp1SpuzpDXJMGZEEII0RjvOSG1sXLlShw6dAgnTpxQ2r1XVlaG0aNHgzGGbdu2qaxn0aJFyM3Nld+Sk5N12WwAgJMT1xNCCCGE1Fe8BiFOTk4QCoVIT09XWJ6eng43N7cq112zZg1WrlyJP//8U+EyyDKyAOTRo0c4c+ZMleelRCIRbGxsFG51ieasJYQQ9UkYQ2R2Ng6mpyMyOxsSPf8S9fb2xvr169UqKxAI8NNPP+m0PfqE1yDE1NQUHTp0QHh4uHyZVCpFeHg4goKCVK63atUqLF++HKdPn0bHjh0rPS8LQO7du4ezZ8/C0dFRJ+2vrV27gDZtgLVr+W4JIYQYhuOZmfC+dAm9bt7E+Ph49Lp5E96XLuF4ZibfTSM1wPvpmNDQUOzatQt79+5FfHw8Pv74YxQWFmLq1KkAgEmTJmHRokXy8t988w0WL16M77//Ht7e3khLS0NaWhoKCgoAcAHIyJEjce3aNezfvx8SiUReRiwW87KNquTnA7duAefP890SQgjRf8czMzEyLg5PSksVlqeUlmJkXBwFIgaI9yBkzJgxWLNmDZYsWYK2bdsiNjYWp0+flierPn78GKmpqfLy27Ztg1gsxsiRI+Hu7i6/rVmzBgCQkpKCn3/+GU+ePEHbtm0Vyly8eJGXbVRFNmlZVBRQh1d7JoQQvcAYQ6FEotYtr7wcn967B2UnXmTLZicmIq+8vNq6NJmZYufOnfDw8ID0lS/pIUOGYNq0abh//z6GDBkCV1dXWFlZoVOnTjh79mzN35RX3Lp1C2+99RbMzc3h6OiI6dOny390A9zcWZ07d4alpSXs7OzQtWtXPHr0CABw8+ZN9OrVC9bW1rCxsUGHDh1w7do1rbVNG/Ri2vaZM2di5syZSp+LjIxUePzw4cMq6/L29tboH4xP7doB5ubA8+fA3btAq1Z8t4gQQupOkVQKqwsXtFIXA/CktBS2akxFXdCtGyyFQrXqHTVqFGbNmoWIiAj07t0bAPD8+XOcPn0av/32GwoKCjBgwAB8/fXXEIlE2LdvHwYNGoSEhAS15qWqSmFhIfr164egoCBcvXoVGRkZeP/99zFz5kzs2bMH5eXlGDp0KD744AMcPHgQYrEYV65cgUAgAABMmDAB7dq1w7Zt2yAUChEbGwsTE5NatUnb9CIIqa9MTIA33gAiIrgp3CkIIYQQ/WJvb4/+/fvjwIED8iDk6NGjcHJyQq9evWBkZISAgAB5+eXLl+PEiRP4+eefVf64VteBAwdQUlKCffv2wdLSEgCwefNmDBo0CN988w1MTEyQm5uLd955B76+vgCAli1bytd//Pgx5s+fjxYtWgAAmjZtWqv26AIFITx7882XQcj06Xy3hhBC6o6FkREKunVTq+z5nBwMuHWr2nK/+fuju4qJKSu+riYmTJiADz74AFu3boVIJML+/fsxduxYGBkZoaCgAF9++SVOnTqF1NRUlJeXo7i4GI+1MBNlfHw8AgIC5AEIAHTt2hVSqRQJCQno3r07pkyZgn79+qFPnz4IDg7G6NGj4e7uDoDLuXz//ffxv//9D8HBwRg1apQ8WNEXvOeE1Hd0MTtCSH0lEAhgKRSqdevr4IAGIhEEquoC4CUSoa+DQ7V1yU5XqGvQoEFgjOHUqVNITk7GhQsXMGHCBADAvHnzcOLECaxYsQIXLlxAbGws/P3962wgxO7duxEdHY0uXbrg8OHDaNasGS5dugQA+PLLLxEXF4eBAwfi3LlzaNWqFU6cOFEn7VIXBSE8e+MNoEkTLhjRs8E7hBCiN4QCATY0aQIAlQIR2eP1TZpAqGGAoQ4zMzMMHz4c+/fvx8GDB9G8eXO0b98eABAVFYUpU6Zg2LBh8Pf3h5ubW7W5i+pq2bIlbt68icLCQvmyqKgoGBkZoXnz5vJl7dq1w6JFi3Dx4kW0bt0aBw4ckD/XrFkzzJkzB3/++SeGDx+O3bt3a6Vt2kJBCM9sbLgL2e3bB5ia8t0aQgjRX8OdnXHUzw+er0w33UAkwlE/Pwx3dtbZa0+YMAGnTp3C999/L+8FAbg8i+PHjyM2NhY3b97E+PHjK42kqc1rmpmZYfLkybh9+zYiIiIwa9YsTJw4Ea6urkhKSsKiRYsQHR2NR48e4c8//8S9e/fQsmVLFBcXY+bMmYiMjMSjR48QFRWFq1evKuSM6APKCSGEEGIwhjs7Y4iTEy7k5CBVLIa7qSm62dnppAekorfeegsODg5ISEjA+PHj5cvXrVuHadOmoUuXLnBycsKCBQu0dhFUCwsL/PHHH5g9ezY6deoECwsLjBgxAuvWrZM/f/fuXezduxdZWVlwd3fHjBkz8OGHH6K8vBxZWVmYNGkS0tPT4eTkhOHDh2PZsmVaaZu2CJihjGetQ5pchlhbJBIgMRGo0MNGCCGvjaou/U4MT1X7U5NjKJ2O0QPZ2YCDAzdENz+f79YQQgghdYOCED1gbw84OnKzpr5IaiaEEPKa2b9/P6ysrJTe/Pz8+G4eLygnRE+8+SaQlMQN1e3Th+/WEEII0bbBgwcjMDBQ6XP6NpNpXaEgRE+8+Sbwv//RfCGEEPK6sra2hrW1Nd/N0Ct0OkZPyCYtu3QJKCvjty2EEKIrNBbi9aCt/UhBiJ5o0YJLTi0qAmJj+W4NIYRol+x0Q1FREc8tIdog24+1PY1Ep2P0hJER0LUr8Msv3CmZTp34bhEhhGiPUCiEnZ0dMjIyAHBzXGg6fTrhH2MMRUVFyMjIgJ2dHYRqXo1YFQpC9Mi77wLt2wO9evHdEkII0T43NzcAkAcixHDZ2dnJ92dtUBCiR0aP5rsFhBCiOwKBAO7u7nBxcUEZJb8ZLBMTk1r3gMhQEEIIIaROCYVCrR3EiGGjxFQ98+wZcPIkcOUK3y0hhBBCdIuCED2zbh0wdCiwYwffLSGEEEJ0i4IQPSObL4QmLSOEEPK6oyBEzwQFAQIB8O+/ACWQE0IIeZ1REKJn7O2B1q25+1FR/LaFEEII0SUKQvQQnZIhhBBSH1AQooe6duX+UhBCCCHkdUZBiB6S9YTExACFhfy2hRBCCNEVvQhCtmzZAm9vb5iZmSEwMBBXqpgkY9euXejWrRvs7e1hb2+P4ODgSuUZY1iyZAnc3d1hbm6O4OBg3Lt3T9eboTUNGwL79wPx8YCFBd+tIYQQQnSD9yDk8OHDCA0NxdKlSxETE4OAgAD069dP5bUFIiMjMW7cOERERCA6OhpeXl7o27cvUlJS5GVWrVqFjRs3Yvv27bh8+TIsLS3Rr18/lJSU1NVm1YpAAIwfDzRpwt0nhBBCXkcCxhjjswGBgYHo1KkTNm/eDACQSqXw8vLCrFmzsHDhwmrXl0gksLe3x+bNmzFp0iQwxuDh4YG5c+di3rx5AIDc3Fy4urpiz549GDt2bLV15uXlwdbWFrm5ubCxsandBhJCCCH1iCbHUF57QsRiMa5fv47g4GD5MiMjIwQHByM6OlqtOoqKilBWVgYHBwcAQFJSEtLS0hTqtLW1RWBgoMo6S0tLkZeXp3DjW0kJsH491yNSXs53awghhBDt4zUIefbsGSQSCVxdXRWWu7q6Ii0tTa06FixYAA8PD3nQIVtPkzrDwsJga2srv3l5eWm6KVpnYgJ8+SVw8CBw6xbfrSGEEEK0j/eckNpYuXIlDh06hBMnTsDMzKzG9SxatAi5ubnyW3JyshZbWTNCIdClC3efhuoSQgh5HfEahDg5OUEoFCI9PV1heXp6Otzc3Kpcd82aNVi5ciX+/PNPtGnTRr5ctp4mdYpEItjY2Cjc9AFNWkYIIeR1xmsQYmpqig4dOiA8PFy+TCqVIjw8HEFBQSrXW7VqFZYvX47Tp0+jY8eOCs/5+PjAzc1Noc68vDxcvny5yjr1UcUghN/0YUIIIUT7jPluQGhoKCZPnoyOHTuic+fOWL9+PQoLCzF16lQAwKRJk+Dp6YmwsDAAwDfffIMlS5bgwIED8Pb2lud5WFlZwcrKCgKBACEhIfi///s/NG3aFD4+Pli8eDE8PDwwdOhQvjazRjp14nJDnj4FHj4EfHz4bhEhhBCiPbwHIWPGjEFmZiaWLFmCtLQ0tG3bFqdPn5Ynlj5+/BhGRi87bLZt2waxWIyRI0cq1LN06VJ8+eWXAIDPPvsMhYWFmD59OnJycvDmm2/i9OnTtcob4YO5OdChA3DpEtcbQkEIIYSQ1wnv84ToI32aJ2T+fG6o7ldfAYsW8doUQgghpFqaHEMpCFFCn4KQrCyuR4SmbyeEEGIINDmG8n46hlTN0ZHvFhBCCCG6YdDzhNQ31GdFCCHkdUJBiAE4fJhLUP3Pf/huCSGEEKI9dDrGAJSWAjExgIEN7iGEEEKqRD0hBkA2adnVq0BxMb9tIYQQQrSFghAD4OMDuLsDZWXAtWt8t4YQQgjRDgpCDIBAQNeRIYQQ8vqhIMRAdO3K/aUghBBCyOuCghADIesJiYoCpFJ+20IIIYRoA42OMRABAUDLlkDbtkBeHmBnx3eLCCGEkNqhIMRAGBsDd+7w3QpCCCFEe+h0DCGEEEJ4QUGIgWEMSEjguxWEEEJI7VEQYkDKyrj5Qlq0AB4/5rs1hBBCSO1QEGJATEyAhg25+1FR/LaFEEIIqS0KQgwMTVpGCCHkdUFBiIGhScsIIYS8LigIMTCyIOTWLSAnh9emEEIIIbVCQYiBcXMDmjThRslER/PdGkIIIaTmKAgxQJQXQggh5HVAM6YaoFGjuKG6gwbx3RJCCCGk5igIMUADBnA3QgghxJDR6RhCCCGE8IJ6QgxUVhawaxdQVAS89RbQrRsgFPLdKkIIIUR9FIQYoOPHgWnTgNxc7vHy5UCDBsCGDcDw4fy2jRBCCFEX76djtmzZAm9vb5iZmSEwMBBXrlxRWTYuLg4jRoyAt7c3BAIB1q9fX6mMRCLB4sWL4ePjA3Nzc/j6+mL58uVgjOlwK6omYQyR2dk4mJ6OyOxsSGrRluPHgZEjXwYgMikp3PLjx2vZWEIIIaSO8BqEHD58GKGhoVi6dCliYmIQEBCAfv36ISMjQ2n5oqIiNG7cGCtXroSbm5vSMt988w22bduGzZs3Iz4+Ht988w1WrVqFTZs26XJTVDqemQnvS5fQ6+ZNjI+PR6+bN+F96RKOZ2ZqXJdEAsyezc0R8irZspAQrhwhhBCi7wSMxy6CwMBAdOrUCZs3bwYASKVSeHl5YdasWVi4cGGV63p7eyMkJAQhISEKy9955x24urriu+++ky8bMWIEzM3N8cMPP6jVrry8PNja2iI3Nxc2NjaabVQFxzMzMTIuDq++wYIXf4/6+WG4s7Pa9UVGAr16VV8uIgLo2VPtagkhhBCt0eQYyltPiFgsxvXr1xEcHPyyMUZGCA4ORnQtpgLt0qULwsPD8e+//wIAbt68ib///hv9+/dXuU5paSny8vIUbrUlYQyzExMrBSAA5MtCEhM1OjWTmqrdcoQQQgifeEtMffbsGSQSCVxdXRWWu7q64u7duzWud+HChcjLy0OLFi0gFAohkUjw9ddfY8KECSrXCQsLw7Jly2r8mspcyMnBk9JSlc8zAMmlpbiQk4Oe9vZq1enurt5rq1uOEEII4RPvianaduTIEezfvx8HDhxATEwM9u7dizVr1mDv3r0q11m0aBFyc3Plt+Tk5Fq3I1Us1mo5gBuG26ABIBAof14gALy8uHKEEEKIvuOtJ8TJyQlCoRDp6ekKy9PT01Umnapj/vz5WLhwIcaOHQsA8Pf3x6NHjxAWFobJkycrXUckEkEkEtX4NZVxNzXVajmAmwdkwwZuFIxAoJigKgtM1q+n+UIIIYQYBt56QkxNTdGhQweEh4fLl0mlUoSHhyMoKKjG9RYVFcHISHGzhEIhpFJpjeusiW52dmggEkFFpwUEALxEInSzs9Oo3uHDgaNHAU9PxeUNGnDLaZ4QQgghhoLXycpCQ0MxefJkdOzYEZ07d8b69etRWFiIqVOnAgAmTZoET09PhIWFAeCSWe/cuSO/n5KSgtjYWFhZWaFJkyYAgEGDBuHrr79Gw4YN4efnhxs3bmDdunWYNm1anW6bUCDAhiZNMDIuDgJAIUFVFpisb9IEQlXnVqowfDgwZAhw4QKXhOruTjOmEkIIMTy8DtEFgM2bN2P16tVIS0tD27ZtsXHjRgQGBgIAevbsCW9vb+zZswcA8PDhQ/j4+FSqo0ePHoiMjAQA5OfnY/HixThx4gQyMjLg4eGBcePGYcmSJTBV89SHtoboAtww3dmJiQpJqi4mJtjWrJlGw3NVSUnhekDKy4G5c2tdHSGEEFIrmhxDeQ9C9JE2gxCAG657IScH8x88wLX8fKxp3BhzGzbUQkuBy5eBN94AXF2BtDStVEkIIYTUmEHME1KfCAUC9LS3xxBHRwBATEGB1ur29weMjID0dApCCCGEGBYKQupQB2trAMD1/Hyt1WlhATRvzt2PjdVatYQQQojOURBSh2RByL/FxcgvL9davW3bcn8pCCGEEGJIKAipQy6mpmggEoEBuKHFUzIUhBBCCDFEFITUsY46OCVDQQghhBBDREFIHetgZQVAu0FIQAD3NzERKC7WWrWEEEKITlEQUsdkeSHXtBiEuLoCly4BOTmAubnWqiWEEEJ0itcZU+ujV5NTrY21swtezO9GCCGEGAzqCaljukpOJYQQQgwNBSE80EVyalISMGsW8OGHWquSEEII0SkKQnigi+TUsjJg82Zg3z7uOjKEEEKIvqMghAe6SE719QUsLYGSEuDePa1VSwghhOgMBSE80MXMqUIh0KYNd5/mCyGEEGIIKAjhAc2cSgghhFAQwhtd5IVQEEIIIcSQUBDCE11O337jBsCY1qolhBBCdIKCEJ7oIjnV3x8wMgJMTLjZUwkhhBB9RjOm8kQXM6eamwMZGYCjY62rIoQQQnSOekJ4oqvkVApACCGEGAoKQniki+RUQgghxFBQEMKjDjpITk1MBPr3B7p311qVhBBCiE5QTgiP5CNktHg6xsYGOH0aEAiAggLgRWcLIYQQoneoJ4RHsp6QhKIirc2c6uICeHhwQ3T/+UcrVRJCCCE6QUEIj2jmVEIIIfUZBSE8o5lTCSGE1FcUhPBMF8mpFIQQQggxBLwHIVu2bIG3tzfMzMwQGBiIK1euqCwbFxeHESNGwNvbGwKBAOvXr1daLiUlBe+++y4cHR1hbm4Of39/XLt2TUdbUDu6SE6VBSG3bgFaSjUhhBBCtI7XIOTw4cMIDQ3F0qVLERMTg4CAAPTr1w8ZGRlKyxcVFaFx48ZYuXIl3NzclJbJzs5G165dYWJigt9//x137tzB2rVrYW9vr8tNqTFdJKf6+gKenkDnzsDz51qpkhBCCNE6AWP8XeosMDAQnTp1wubNmwEAUqkUXl5emDVrFhYuXFjlut7e3ggJCUFISIjC8oULFyIqKgoXLlyocbvy8vJga2uL3Nxc2NjY1LgedXlFR+NJaSn+atsW3e3stFInY9wwXUIIIaQuaXIM5a0nRCwW4/r16wgODn7ZGCMjBAcHIzo6usb1/vzzz+jYsSNGjRoFFxcXtGvXDrt27apyndLSUuTl5Snc6pIuklMpACGEEKLveAtCnj17BolEAldXV4Xlrq6uSEtLq3G9Dx48wLZt29C0aVP88ccf+Pjjj/Hpp59i7969KtcJCwuDra2t/Obl5VXj168JXSSnypSWar1KQgghRCt4T0zVNqlUivbt22PFihVo164dpk+fjg8++ADbt29Xuc6iRYuQm5srvyUnJ9dhiysEIVpMTn3wAGjaFGjYkDs1QwghhOgb3oIQJycnCIVCpKenKyxPT09XmXSqDnd3d7Rq1UphWcuWLfH48WOV64hEItjY2Cjc6pIuklPd3YGkJCAjA0hN1UqVhBBCiFbVKAjZu3cvTp06JX/82Wefwc7ODl26dMGjR4/UqsPU1BQdOnRAeHi4fJlUKkV4eDiCgoJq0iwAQNeuXZGQkKCw7N9//0WjRo1qXKeuuepg5lRzc6BFC+4+zRdCCCFEH9UoCFmxYgXMzc0BANHR0diyZQtWrVoFJycnzJkzR+16QkNDsWvXLuzduxfx8fH4+OOPUVhYiKlTpwIAJk2ahEWLFsnLi8VixMbGIjY2FmKxGCkpKYiNjUViYqK8zJw5c3Dp0iWsWLECiYmJOHDgAHbu3IkZM2bUZFPrDM2cSgghpN5hNWBubs4ePXrEGGPss88+YxMnTmSMMXb79m3m5OSkUV2bNm1iDRs2ZKampqxz587s0qVL8ud69OjBJk+eLH+clJTEAFS69ejRQ6HOX375hbVu3ZqJRCLWokULtnPnTo3alJubywCw3Nxcjdarja+SkhgiItiEuDit1bl6NWMAY6NGaa1KQgghpEqaHEONaxK4WFlZISsrCw0bNsSff/6J0NBQAICZmRmKi4s1qmvmzJmYOXOm0uciIyMVHnt7e4OpkWX5zjvv4J133tGoHXzTRXKqrCfkxg2tVUkIIYRoTY2CkD59+uD9999Hu3bt8O+//2LAgAEAuGnVvb29tdm+euPV5FRr4xrtGgUBAdzfxEQgPx948RKEEEKIXqhRTsiWLVsQFBSEzMxMHDt2DI6OjgCA69evY9y4cVptYH2hi+RUZ2egXz9g2jSgsFArVRJCCCFaw+u07fqqrqdtlxl66xZOZmVhna8v5tTxhGmEEEKINuh82vbTp0/j77//lj/esmUL2rZti/HjxyM7O7smVRLoduZUQgghRN/UKAiZP3++/Poqt27dwty5czFgwAAkJSXJk1SJ5nSRnAoAYjHw779arZIQQgiptRplPyYlJclnJT127BjeeecdrFixAjExMfIkVaI5XSSnJicDvr6AkRFQUABooUpCCCFEK2rUE2JqaoqioiIAwNmzZ9G3b18AgIODQ51fgfZ1UjE5NVZLvSGenoCZGXchu1cmkiWEEEJ4VaMg5M0330RoaCiWL1+OK1euYODAgQC46dEbNGig1QbWN7KZU69pKS/EyOjlUF2aL4QQQog+qVEQsnnzZhgbG+Po0aPYtm0bPD09AQC///473n77ba02sL7RRXIqTd9OCCFEH9UoQ6Bhw4b49ddfKy3/9ttva92g+k6XM6dSEEIIIUSf1DhNUSKR4KeffkJ8fDwAwM/PD4MHD4ZQKNRa4+ojXSSnVgxCGAMEglpXSQghhNRajY5wiYmJGDBgAFJSUtC8eXMAQFhYGLy8vHDq1Cn4+vpqtZH1iaupKTxNTZEiFiO2oADd7OxqXaefHyAUAllZQEoKQGk7hBBC9EGNckI+/fRT+Pr6Ijk5GTExMYiJicHjx4/h4+ODTz/9VNttrHc6vugN0VZyqpkZMHs2sGIFYGqqlSoJIYSQWqtRT8hff/2FS5cuwcHBQb7M0dERK1euRNeuXbXWuPqqg7U1TmZlaTU5de1arVVFCCGEaEWNekJEIhHylRwgCwoKYEo/tWtNVzOnEkIIIfqkRkHIO++8g+nTp+Py5ctgjIExhkuXLuGjjz7C4MGDtd3GeufV5FRtkEqB+/eBU6e0Uh0hhBBSazUKQjZu3AhfX18EBQXBzMwMZmZm6NKlC5o0aYL169druYn1jyw5VZszpz5/DjRpArzzDkCT2hJCCNEHNcoJsbOzw8mTJ5GYmCgfotuyZUs0adJEq42rzzpaWyPlRV6INkbIODlxo2KePAFu3gS6dat9GwkhhJDaUDsIqe7quBEREfL769atq3mLCICXyanaGiEDcPOFPHnCzRdCQQghhBC+qR2E3FDzwiMCmglLK3Q1c+qvv9LMqYQQQvSD2kFIxZ4Oonu6mDm1XTvuLwUhhBBC9EGNElOJ7ukiOVU2ffvt20BZmVaqJIQQQmqMghA9pu0r6np7AzY2gFgM3L2rlSoJIYSQGqt9Hz/RmY7W1vhZi8mpRkbAqlWAgwPg5aWVKgkhhJAaoyBEj+kiOfXDD7VWFSGEEFIrenE6ZsuWLfD29oaZmRkCAwNx5coVlWXj4uIwYsQIeHt7QyAQVDs52sqVKyEQCBASEqLdRtcBXcycSgghhOgL3oOQw4cPIzQ0FEuXLkVMTAwCAgLQr18/ZGRkKC1fVFSExo0bY+XKlXBzc6uy7qtXr2LHjh1o06aNLpquc7pIThWLgT/+4C5ox5hWqiSEEEJqhPcgZN26dfjggw8wdepUtGrVCtu3b4eFhQW+//57peU7deqE1atXY+zYsRCJRCrrLSgowIQJE7Br1y7Y29vrqvk6p+3kVKmUm7p93jxu4jJCCCGEL7wGIWKxGNevX0dwcLB8mZGREYKDgxEdHV2rumfMmIGBAwcq1K1KaWkp8vLyFG76Qtt5IWZmQMuW3H2aL4QQQgifeA1Cnj17BolEAldXV4Xlrq6uSEtLq3G9hw4dQkxMDMLCwtQqHxYWBltbW/nNS4+GjnR8EYRoe/p2gIIQQggh/OL9dIy2JScnY/bs2di/fz/MzMzUWmfRokXIzc2V35KTk3XcSvXpIjmVghBCCCH6gNchuk5OThAKhUhPT1dYnp6eXm3SqSrXr19HRkYG2rdvL18mkUhw/vx5bN68GaWlpRAKhQrriESiKvNL+CRLTk0RixFbUKCVK+pSEEIIIUQf8NoTYmpqig4dOiA8PFy+TCqVIjw8HEFBQTWqs3fv3rh16xZiY2Plt44dO2LChAmIjY2tFIAYAm0npwYEcH8fPAByc7VSJSGEEKIx3icrCw0NxeTJk9GxY0d07twZ69evR2FhIaZOnQoAmDRpEjw9PeX5HWKxGHfu3JHfT0lJQWxsLKysrNCkSRNYW1ujdevWCq9haWkJR0fHSssNRYcXM6dqKznV0ZGbMTU5GfjnH6BbN61USwghhGiE9yBkzJgxyMzMxJIlS5CWloa2bdvi9OnT8mTVx48fw8joZYfN06dP0U52OVgAa9aswZo1a9CjRw9ERkbWdfPrhC6SU7/7DnByAvz8tFYlIYQQohEBYzRl1avy8vJga2uL3Nxc2NjY8N0cpIvFcLt4EQIAuW++CWtj3mNHQgghRClNjqGv3eiY15EuZk4lhBBC+EZBiIHQdnKqWAxs2QJMn87dJ4QQQuoaBSEGQtszp5qYAJ9/DuzaBdy9q5UqCSGEEI1QEGIgtN0TIhDQfCGEEEL4RUGIgehgZQUAuEszpxJCCHlNUBBiINxEIq0np1IQQgghhE8UhBgQbZ+SqRiE0EBtQgghdY2CEAOi7eTUVq0AY2MgO5ubPZUQQgipSxSEGBBt94SIRFwgAgDx8VqpkhBCCFEbTb1pQF5NTtXGzKlHjwKuroAeTAxLCCGknqGeEAOii+TUpk0pACGEEMIPCkIMjLZPyRBCCCF8oSDEwGg7OVUiAebMAXr2BHJytFIlIYQQohYKQgyMtntChELg+HHgr7+Amze1UiUhhBCiFgpCDIwuZk5t1477S5OWEUIIqUsUhBgYmjmVEELI64KCEAOky5lTCSGEkLpCQYgB0nZyqiwIiYsDxGKtVEkIIYRUi4IQA6TtnpBGjQBbW6CsjGZOJYQQUncoCDFAFZNTC7SQnCoQcL0hLi5AenqtqyOEEELUQtO2GyA3kQgepqZ4KhbjRkEButnZ1brOU6cAS8vat40QQghRF/WEGKiOWj4lQwEIIYSQukZBiIFq9+KUzNHMTERmZ0PCGM8tIoQQQjRDQYgBOp6ZiS1PnwIAovLy0OvmTXhfuoTjmZk1rpMxYMQIwMsLePRIWy0lhBBCVKMgxMAcz8zEyLg4PCsrU1ieUlqKkXFxNQ5EBALg/n3gyROaL4QQQkjdoCDEgEgYw+zERCg78SJbFpKYWONTMzRpGSGEkLqkF0HIli1b4O3tDTMzMwQGBuLKlSsqy8bFxWHEiBHw9vaGQCDA+vXrK5UJCwtDp06dYG1tDRcXFwwdOhQJCQk63IK6cSEnB09KS1U+zwAkl5biQg0vhyu7hsyZM8DBg0BkJHeVXUIIIUQXeA9CDh8+jNDQUCxduhQxMTEICAhAv379kJGRobR8UVERGjdujJUrV8LNzU1pmb/++gszZszApUuXcObMGZSVlaFv374oLCzU5aboXKqa05mqW+5VeXnc36goYPx4oFcvwNubu8ouIYQQom0CxvgdVhEYGIhOnTph8+bNAACpVAovLy/MmjULCxcurHJdb29vhISEICQkpMpymZmZcHFxwV9//YXu3btX26a8vDzY2toiNzcXNjY2am+LrkVmZ6PXzZvVlosICEBPe3uN6j5+HBg5kktQrUgg4P4ePQoMH65RlYQQQuohTY6hvPaEiMViXL9+HcHBwfJlRkZGCA4ORnR0tNZeJzc3FwDg4OCg9PnS0lLk5eUp3PRRNzs7NBCJIKiijKepqcaTl0kkwOzZlQMQ4OWykBA6NUMIIUS7eA1Cnj17BolEAldXV4Xlrq6uSEtL08prSKVShISEoGvXrmjdurXSMmFhYbC1tZXfvLy8tPLa2iYUCLChSRMAUBmIuJuaVhmkKHPhAjcqRhXGgORkrhwhhBCiLbznhOjajBkzcPv2bRw6dEhlmUWLFiE3N1d+S05OrsMWama4szOO+vnBUyRSWO5qYgJjANcKChD2+LFGdaamarccIYQQog5erx3j5OQEoVCI9Feumpaenq4y6VQTM2fOxK+//orz58+jQYMGKsuJRCKIXjmo67Phzs4Y4uSECzk5SBWL4f7iFMyetDS8n5CAxUlJ6GBlhbcdHdWqz91dvddVtxwhhBCiDl57QkxNTdGhQweEh4fLl0mlUoSHhyMoKKjG9TLGMHPmTJw4cQLnzp2Dj4+PNpqrV4QCAXra22Ocqyt62ttDKBDgPXd3THd3BwMwPj4eScXFatXVrRvQoMHLJNRXCQTcTKrdummv/YQQQgjvp2NCQ0Oxa9cu7N27F/Hx8fj4449RWFiIqVOnAgAmTZqERYsWycuLxWLExsYiNjYWYrEYKSkpiI2NRWJiorzMjBkz8MMPP+DAgQOwtrZGWloa0tLSUKzmQdmQbWzaFJ2trZFdXo7hcXEoUiObVCgENmzg7qsKRNav58oRQggh2sL7EF0A2Lx5M1avXo20tDS0bdsWGzduRGBgIACgZ8+e8Pb2xp49ewAADx8+VNqz0aNHD0RGRgIABCqOpLt378aUKVOqbY++DtFVV3JJCTpcv47MsjJMcnXFnhYtVL4nFR0/zo2SeTVJ1c8PuHmTghBCCCHV0+QYqhdBiL4x9CAEACKysxF88yakALY0bYpPPD3VWk8i4UbBpKZy9z/8ECgqAr7+Gvj8c922mRBCiOGjIKSWXocgBADWPH6M+Q8ewEQgQGTbtuhia6txHXv3AlOmcL0gFy4AtUjV0aqKwZK7O5evQj01hBDCP4OZrIzo1lwvL4xydkYZYxgZF4e0Kq47o8qkScC4cdxBf/x44MW8b7w6fpybTr5XL5penhBCDBkFIa8xgUCA75o3RysLC6SKxRh95w7KpFIN6wC2beMO8oxVPalZXZBNL/9qO1JSuOUUiBBCiOGgIOQ1Z21sjOOtW8NaKMSF3Fx89uCBxnXY2gKnTgGxsVySKl9oenlCCHm9UBBSDzS3sMC+Fi0AAOufPMHBVyaHU0erVkDFS9LwkUlE08sTQsjrhYKQemKoszM+b9gQAPB+QgJuFRTUqB7GgB07gB49ALFYmy2sHk0vTwghrxcKQuqRr3x80NfeHkVSKYbdvo2csjKN63j2jBuqe+EC8MUXOmhkFWh6eUIIeb1QEFKPCAUCHGjVCo1EItwvKcHEu3ch1fC8irMz8N133P3Vq4EzZ3TQUCUYA86fr7oMTS9PCCGGhYKQesbRxATHW7eGmZERfs3Kwv89eqRxHUOHAh99xN2fNAnIzNRuG5XZsQNYuvTl41cngJU9punlCSHEcFAQUg+1t7bGtqZNAQBfPnyI37KyNK5j7VouWTUtDZg6VfeJqhMnchOlbdkCHDsGvDoBLGPAggXA8OG6bQchhBDtoRlTlXhdZkytzsf//ovtT5/CztgY1zp0gK+5uUbr37oFdOoElJZyF8D79FPtti8nhxseLOvlKC8HjI25+xVnTP3lF+DgQaBNG+DGDcCIQmtCCOENzZhK1LK+SRO8YWODnPJyDL99W60r7lbk78/1iBgZATUcbKNSQgLQrh2wfPnLZbIABOBOufTsyc3munkzYGMD/PMPcOSIdttBCCFEdygIqcdERkY46ucHFxMT/FNYiOkJCSiXShGZnY2D6emIzM6GpJqOsk8+4SYx0+bF7aKigC5dgIcPgf/9r/oAx8EBmD+fu79kCddjQgghRP/R6Rgl6svpGJm/cnLQOzYWEgB2QiFyKvSINBCJsKFJEwx3dlarroqnTGri2DFgwgTuFE9gIHeqRZ2Xzs8HGjfmhhD/97/Ae+/VvA2EEEJqjk7HEI30sLPDRFdXAFAIQAAgpbQUI+PicFyNITB373I5IkeP1qwdGzYAo0ZxAcjgwcC5c+oFIABgbQ0sWsTdX7aMq4MQQoh+oyCEQMIYzmZnK31O1k0WkphY7amZH37gTs188AHw+LFmbViwgLvuC2PcKZ7jxwELC83q+PhjwMMDyMoCrl/XbF1CCCF1j4IQggs5OXhSxRzsDEByaSku5ORUWc/SpUDnztyolnff1exCci8ubYNvvuESTWsy14e5OZeYev8+l1NCCCFEv1EQQpCq5kVgfnv+HOVSqcrnTUyAAwe4UyMXLgBff61+G6ZO5Yb8fvZZ5YnINNG1K+DmVvP1CSGE1B0KQgjcTU3VKrc6ORkNoqMxNzER/6gYsuLrC2zdyt1ftowb6aLMw4fAO+8AGRkvl7VurUGj1XDhAtcrQwghRD9REELQzc4ODUQiVNUBYSUUwtHYGOllZVj35AkCrl1Du2vX8G1yMtJf6Ul5913uJpUC48cDGc8Y1kdmY9aJdKyPzEb0ZYY33gBOneLyOHRh1iyge3duHhNCCCH6iYboKlHfhugCwPHMTIyMiwPwMhkVgDwwOernh3ccHXH6+XPsTUvDL1lZKHvxryME8LaDAya7uWGQoyPMhELk5XGTjRV3ykT6qERIHSsMV8kQAZuboE2uM377rfIU7FrZnuPAiBGApSXw4AHg4qL91yCEEFKZJsdQCkKUqI9BCMAFIrMTE/GkwvhWL5EI65XME5JVVobDGRnYm5aGK/n58uV2xsYY4+yMyW5u2PNrKXZ63OGeqNjNIuUef5rlhw0j1RyDqyHGuOHC168Dc+YA69bp5GUIIYS8goKQWqqvQQjADde9kJODVLEY7qam6GZnB2E1maJ3CwuxLz0d/0tPVwhgIAF3wk/Z6lJAmC1C0ZA3YGpci0zUKpw+DfTvD4hEQGIi0KCBTl6GEEJIBRSE1FJ9DkJqQ8IYInNysDctDYfSM1CG6v+1vkUAQnra66Q9jAE9enAJqh9+CGzfrpOXIYQQUgHNmEp4IRQI0NveHvtatsTUnGZqrXM/W73hwTUhELwcJvzdd9z8IYQQQvQHBSFEJ1ramalVztdeveHBNdWtG9CvH3cqJiVFpy9FCCFEQ3oRhGzZsgXe3t4wMzNDYGAgrly5orJsXFwcRowYAW9vbwgEAqxfv77WdRLt++RNOwizRFwSqjIMMMo1wSdv2um8LXv2AAkJ3JBdQggh+oP3IOTw4cMIDQ3F0qVLERMTg4CAAPTr1w8ZFWexqqCoqAiNGzfGypUr4aZiakxN6yTaZ2osQKhpEy4p9dVAhAEQAMymDBueJkPXaUluboCa87ERAyCRAJGRwMGD3F9NLg9ACNEvvCemBgYGolOnTti8eTMAQCqVwsvLC7NmzcLChQurXNfb2xshISEICQmpVZ2lpaUorTCqIy8vD15eXpSYqgWf/ZKJdeJESCrMEyJ8bopm1maIN8kDAAx2dMSeFi1gb2Ki07aUlXG9Ih07cnOYEMNz/Dgwezbw5MnLZQ0acFdgHj6cv3YRQl4ymMRUsViM69evIzg4WL7MyMgIwcHBiI6OrrM6w8LCYGtrK795eXnV6LVJZasGOaNoyBv4FgGYmd0S3yIARYODEBfcDtuaNoWpQICfs7LQ/vp1XK8w34guLFwITJ8OLFqk05chOnL8ODBypGIAAnC5PiNHcs8TQgwLr0HIs2fPIJFI4OrqqrDc1dUVaWlpdVbnokWLkJubK78lJyfX6LWJcqbGAoT0tMemYa4I6WkPU2MBBAIBPvL0RHT79vAxM8PDkhJ0iYnB1pQUnZ2emTEDMDYG/viDG7ZLDIdEwvWAKPvXkC0LCaFTM4QYGt5zQvSBSCSCjY2Nwo3UjfbW1ojp0AFDnZwgZgwz7t3D+Ph45JeXa/21GjcG3nuPu/+f/yg/oBH9dOFC5R6QihgDkpMpuCTE0PAahDg5OUEoFCI9PV1heXp6usqkUz7qJLplZ2KC435+WOvrC2OBAIcyMtDp+nXcVnGl3tr44gtuBtULF4A//9R69URHUlO1W44Qoh94DUJMTU3RoUMHhIeHy5dJpVKEh4cjKChIb+okuicQCBDq5YXItm3haWqKhOJidI6Jwd4anpZTpUED7rQMQL0hhsTdXbvlCCH6gffTMaGhodi1axf27t2L+Ph4fPzxxygsLMTUqVMBAJMmTcKiCpmEYrEYsbGxiI2NhVgsRkpKCmJjY5GYmKh2nUR/dbW1xY2OHdHP3h7FUimm3L2L9+/eRbEWT/YvXAhYWXEXtztxQmvVGjQJY4jMzsbB9HREZmdDomfRWbdugLV11WXs7IA336yT5hBCtMSY7waMGTMGmZmZWLJkCdLS0tC2bVucPn1anlj6+PFjGBm9jJWePn2KdhXGV65ZswZr1qxBjx49EBkZqVadRL85m5ritzZt8PWjR1j68CG+S0vD1fx8HPXzQ1MLi9rX78wlMV68CDRqVPv2GjplV09uIBJhg5KrJ/MlPByobvBUTg7wwQfcNYJEojppFnkNSCTc6dnUVK4nrVs3QCjku1X1B+/zhOgjuoCd/gjPzsb4O3eQUVYGa6EQ3zVvjlEuLjW62m9F5eXcSJn67nhmJkbGxVW61KDsnTzq58d7IJKWBgQEABkZ3BT8cXGKSapeXkDfvtwcMBIJEBTEDdelFDD+GMqBnead0Q26im4tURCiX56WlmLcnTs4n5sLAOjv4IB/CgqQIn558Tt9++VuCCSMwfvSJYUekIoE4N7XpDfe0CjAe/U1ahMsAi97OBISgMuXudlvlR3gzp4FRo3iyjdoAJw8CbRvX6Nmk1owlAO7bN6ZV4+Asn/Po0f1q72GhIKQWqIgRP+US6X4IikJ36iYw6Wmv9yfPQNWrgT8/ID6ljIUmZ2NXjdvVlvuXVdXvOPoiNaWlmhmbg4TI/VSybR5mocxLriwt6+63L17wODBwN27gIUF99jDQ6OXIrVgKAd2iQTw9lY97Fsg4AKnpCT97MHRdxSE1BIFIfpJwhhco6KQpWIOkZr8ct+0Cfj0U8DTE0hMBMzUu/ivwcstL8en9+5h3ytD2atjIhCguYUFWltaorWlJfxf/PU2M4NRhfdcG6d50tMBF5eXBzB15eYC48dz0/MvW6bZuqTmDOnAHhkJ9OpVfbmICKBnT1235vWjyTGUzooTg3EhJ0dlAAJw18VLLi3FL8+eYaiav7SnTwdWr+Ymutq2DZgzR0uNfUEbpyO06Z+CAmxJScEP6ekokqq6xLGigQ4OyCovx+3CQhRIJLhdWIjbhYUKZSyMjOD3IiBpZWGBb5KTKwUggPzahQhJTMQQJyeV70VODhAYCHTuDOzaBdjaqr+NtrbAzz8rBi9padyIKCsr9eshmtFkQjm+D+w074z+oCCEGIzUCjkgVRkWF4c2lpbobW+P3vb26G5rC2sVWagiEbBkCZdz8H9hDKUtc5BSLIavvSk+edMOpsY1Dxj0ZdSJWCrF8cxMbHn6FH+/yKsBgFbm5nhaVobc8nKlAYOsZ+mkvz+EAgEYY3hcWioPQmS3+MJCFEmluJqfj6tqXP9HFixeyMlBTyXnVxjjgsNHjwA1z/xUUvGXdnEx8M47gFjMBSfe3jWrk1TNkA7sNO+M/qDTMUrQ6Rj9pG4Ow6uMBQJ0traWByVv2NhAVOHoVlYGuIzMRM6ERMClwtV+s0QINW2CVYM0Dxh0OepE3d6VJyUl2Jmail2pqUh7EcAZCwQY5uSEGZ6e6G5rixPPnmFkXBwAKLRVk3aWS6VILC6WByW/P3+OK2oEIwdatsQ4JcPmd+4EPvyQG730999cj0htxMdzXe/p6YCTE3DsGNC9e+3qJIqSk4GJE4G//qq+rD6c4igsBBwcuMBUGX06dWSIKCekligI0U+y0RwppaVV/nK/3L49zufmIjw7G+HZ2XhQUqJQztzICN1sbeVBycHIYqy1uvOyEhkp93h+gZ9agQhjDPkSCTLFYnS5cQMZZWVKy9Vm1El1vSuMMUTk5GBLSgpOPnsG2RRv7qammO7ujukeHvB4ZRINZXV6iURYX8MeG3WDxT3Nm2PyKz814+K4XI6SEuCbb4DPPtP45ZV68gQYMgSIieGCm61bud4vUnvZ2VzvUl5e9WXd3Lh9weeBnTFg2jRuSDfABRzKjoLHjulHEq0hoiCkligI0V+yHgZA/V/uD4uLEZ6Tg/DsbJzLzkb6q8HBi2ADyuIBBhjlm2BHBx/kSyV4Xl6O7LIyZJeXy+8/Ly9H9ov7mszr2tnaGp1tbOBrZgZfc3P4mpvDx8wM5iq+oavrXZnm5oaLeXmILyqSP9fD1hafeHpimJNTlaNatJm7Ul2wWFF/BwfM9fLCW3Z2KC4WoHNnLhDp2xf4/fean45RpqiIO/gcPsw9njkT+PZbLigxlHkt9FVICHDtGjBsGDB/PrdM2ZHF3Z3bv9WNctKlrVu5SzcYGXGXbti9u3Iui7MzcP9+9bP0EuUoCKklCkL0W21+uTPGEFdYKA9K/szKRinUS9BUlzGA2lwD2MPUVB6UyAIUbzMzjIyLw1M18mKshEJMdHXFJx4eaM1TJmZVwSIDF4Bdzc+XPxdgaQmHc16I+MIFro5GuHkT0MUEx4wBK1ZwFzIEuIPRW28ZxrwW+uLJE2DePO49bN2aW1ZSws3fYmSkfJ4QDw+gtBTIygJ69+YCTBMTftr/6BEwdCgwaRKXiF4xALW1BT7+GHj8mDsluH07P200dBSE1BIFIfpPW7/cZ5xIw1b7u9UXTLQEHlkC+cYQFJjASWSMbm2NMWmYCRyMjWFvbAyR2AQNbI1xOS9PrdMRcxs0gLFAgPslJbhfXIz7xcXIq+U1cj718MDyxo1howfTwVYXLN4vLsb6J0/wfWqqfKSOUbYpJtt4Yk13Dzjo8Ch18iR3APrsM+CTTwAmYIB/DuAoBrJMgdt2EEgFejOvhYwuRlupW6dYDKxfD3z1FZdT0bs3N0Gc0jqV9Czdvg107cqt+9FHXI8EXwPFSkq4pHRlrx8RwQWmAPDHH1yv3OtKVz2AFITUEgUh9cf6yGzMQfUBQ8f9ARDctEd8PFBQwC2bOxdYs4a7n5nJzWnh5QU0b8kQ/v4lMMdS5ZeIlALCbBGKhryhMPqGMYassjI8qBCUyAKUWwUFyFEjQFGV7MkXdQ5wz8vKsPPpU2xKSZH39FgYGWGqmxtCGjRAk1euF6StA3FJCdC0KfDEJxOYqZiUjAwRsKUJvJKc9SY5URejrdSt8+xZYNYsbhI4gAsmtmzhptPXxM8/c70QjAEbN3J11oX8fO5aUf36qVd+1ixg82auR+z2bc2GiBsKXc5sS0FILVEQUn+IyxksTl6CxF69gIExICWFG3Hh4cHNtAoAUVGvXMG1WyawLI4792CkWB8EAJb6YV6gM4KDuQ9+gwaAjY3qX4bqJntGBAQoHfaqr8rKuK5vX19uKPHhjAysTU7GzRfzkAgADHFywtwGDdD1xWgebR2IIyOBXkte7CfZi8lU2E8RXznzPppDF6Ot1Kmzc6kz5s4Fjhzhlrm4cPPqTJxY816M1au5HigjI+DUKeDtt2tWj7qkUu6gevKk+oFPYSEXYN2/z82k/P33um1jXdP1zLYUhNQSBSH1y2e/ZGK1leqAQd3RMc+fc8HJnj3Af/8LLhB59Rd2OvcLGxcq12dlxQUjoaEvR27k5XHdpS5uDG88vASpmsGSofj8c+7AsHMnN8spwPUIncvJwbrkZPz2/Lm8rK+ZGe6/MtIJ0OxALGUMBRIJ8iUS7P+5DAuMbwJ2ZcqTkqUAMkUYuP8N7P5OAE07G7TV1V3dNX4AwMnEBP9r0QJWQiHMhUKYGxnB3MgIFhXuV0xMltdZUqoyIdvLTISQy29g7hwBjIy4RN5lywA7O823QaHqF6NTzp8Hfv0VaNmydvVVZ/Fi4P/+jzv9cv48NwGeOv7+m7sW0bZtXO8N37TVA1gXM9tSEFJLFITUP5/9kol14kRIHGs/T4jClNBGr+Qa3LIDpNwXR2AgN5HWkydcACOzaRP3hQ8A0dFAly4vnlCjd0UffrWr6+xZ7nw7Y9wv7VGjKpe5U1iIb588wb7UVFSXkmstFGK8iwsKpVLklZcj/0WwUfF+QU1ybpa2RO4vrpB9Fezbx805EhgIdOgAWFpWXuX4ceDTEIYUhxz5vvd8boeN6wVq/8IUS6W4UVCAvWlp2Pb0qebtfoUQkAcoRkDlUWJKnGkdgAOf2WP2bM1PvVSltJQ7renoqL06lfnxR2D0aO7+3r1cMqomiosBc3Ptt0tT2jwVVxdT1lMQUksUhNRP4nKGrX/n4H527WZMlf3SSElRPkxR2S+NoiKu/JMn3KmJhg255VFR3LVtEhNfzMNQTe/KgQPAuHEaN7nOZWRwB7W0NG521B07qi5/IjMTw1+MttEGIQCRkZHaU9d7iUToYmODLra2+G6uDf45ZgVIjCAUciNEAgNf3u7eBUZuzARmKM8zOTbbWWkgkl1Whot5eYjKzUVUbi6u5OejRM32ydpoZmSEYqkURRIJiqVSFGuwvjLrGzfB7IYNalWHOq5cAdq25UbYaEtsLJe7UlTE9S6uXVtN4UWLgLAwriFK5Obykxui7VNxBw++7HWsSm2+SygIqSUKQkhtyc65AoqBSE3Puarbu6IPs1FWRyoFBg4ETp/mcmquXOGueFuVg+npGB8fX23dw52cEGRjAxtjY1gLhbAWCpXeNzMywl85OWrl2RgBlQZxC8uMIEy0hjjGFoiz4W55pnB2BqRdM5H1qeo8E8dNfkg74oSH4mJEVQg67lSY30XGwdgYzSwscEmNmcCU5QMxxlD6IhgpkkpR/CI4+f5aLtbjXrV1AkA7KysMcXLCEEdHBFhZQaDlIS179nCnHydOBL77TjsjZjIzgU6duOG4fftyuSdVDhiTnbNZvJgb/vOKw4e5UVTffVe3p2aqOxVXk4kP6+K7hIKQWqIghGiDsuxzLy9umKOmSV/q9K54eHBfuvowkqMqa9ZwE1qZmQFXr76ca6IqukjMVTcv4nanTojJz8fFvDxczM3Fxbw8ZCu5kKL5M3PYPLFBum8WYFWusk6UGsHOzAg5SmaTaWpujq62tuhqY4OutrZobmEBBqg1U7AmB6L9BxneFV4CnFTkGDEAZQLAhClsRyORCIOdnDDUyQndbG1VToCnSf7C779z1/aRSrn/jblz1dqEKskmJGvaFLh8WY3J0dq2BW7e5P7euFHp6UWLgJUrucTcuDhu+v+q1CZ/I620FDEFBYjJz8cfz5/jbzUC0CGOjujn4AA/S0v4WVrCscLw9vJyrvfj0SNubhd5TogOR4VREFJLFIQQbdHmOPyqelcY4yb3unwZaNSo9u3WlX/+4fIoysu5iaA+/FC99dSdsl/TqfBrMgOvlDH8W1SkEJTEK+nFqI6pQIAO1tbyoKOLrS1cVJyPqEk7q/LLL8Dg1dXnGJ34yhbZzbNw8tkz/JmdrXB6x87YGAMdHDDEyQlvOzjILxJZk/yFDRu4WVcFAm4Uy6BBam+KSt9/DwQFqZH4mp7OzSdf8bGLi0KR0lLu/zYujssxkc26q4y6288YQ3JpKWLy8+VBR0xBgdoX6qyKm6kpWplbQJhsidifLJF5xRLGTyyReNMYjRpVSMYHanWpClUoCKklCkKIvlLWu+LpyR3U09O5npZz54AmTfhrY1XKyoAvv+RyXA4d0qzrXdsH4or11vbaOc/LynApLw8rbj1FFLKqLd/+ViOcmdoQDlbqR6TavMaPRMIl1JZ2VpFjtLXyr+EiiQRns7Nx8tkz/JKVhcwKia2mAgHesrdHQ5EIu1JTNc5fYIybqXTHDm6UWFQU0KbNK21Wo3eBsRqcztm3D5g8WfHxxImVil2/zuX8SCRcECJLeK2ouvyNOS8mKLzxIujIUtKjZgSghYUF2ltbw0YoxFY1kpLHOjsjXyLB7cJCPKpiFJWniQh+lua4mJ+PgnJJlT2ANbm2lQwFIbVEQQjRZ8p6V9LSuBksExK4ZWfPAq1a8d1S1T1BUmnNrguj7YvtyduppeGP4VnZCL6lxpWeQwLw10Z7ja/mW9N23rzJpTps3fpyOvydO7mZS5kRA1rnaDRbrIQxXMrLw8lnz/DTs2e4V1xcbRuq660qK+PmDDl3jkvMvnLlZVvV6V04fx74+mtg//7qT5dUJB0zBjh2DEYSCaRCITByJIwOHVJadulS7n10dOR6RSrOC6jOUOpXGQsEaG1pifZWVmhvbY32VlZoY2UFyxeRn6Y9gFevAiMmliNZUAT4FMK8ZSG8uheiwKkQT8s062GpzZxDFITUEgUhxBClpwN9+gC3bnFfwmfOqEz0rxO6mpFRF1OXa4uEMbieu4Qsger5XKxKRBh14g18v+tlm3UxOgQAHj7kci337+d6CWbM4GYCldFG3hJjDHeLivBtcjJ2paVVWz7Y3h7trazgKRJxN1NTeIhEcDc1RX6OEd54A7h3jwsoPv9cvdEhHYqc0akTl5AaEsJdmFAuJYX7cChx7vlzdBoyBNYVTqflWVri2k8/4S0Hh0rly8qAgdNcceaOJ4YMY1j5v2LEFRXiVkEBIrKzcV6N/I1Bjo4Y5OiI9tbWaG1pCVE10bgmPYBZWVy+h5UVNyHchx++TPrOKStDXFERdqem4js19lNtZl+mIKSWKAghhiori5ua+vp1blKpP/5Qf3ImbdL1jIz67HhmJkbcVp1rcay14imJjAzAx4dLQ1i6FHj33WpGcqjh2TPuIL51K3fNFwAYM4YbAPLqqTpt5S2pO4JJFQEAFxMTOEEEaaYI3VqawsPUFBtSUpQmAsvW8TQVwXHmG7h5Q4B27bhJxhRGW/XuzXWvqCAVCGBU4R/11ceviu3wBtp/uAlGjQshEWp++KzJwV3VPEY9bzeB811uaL7ssxUVBbRvr3p+k7qYfZmCkFqiIIQYstxcYMAA7loZO3e+nH21rtTFjIz67nhmJmbfS8QTcfXJmbKZOWU/Tps144KRMWNq9v6sXg0sX85dLwXgjsHffMMlVuqSuge36e7uMDMywlOxGCmlpUgpLcVTsRjltTkUxdrA7LkFJgwzhpe9MWyFQtgaG8PW2BhNf/0VLefMgTA3V2kKhLoYgGwrK3w4dy6Ovhi7am5kBD9LS7S2tIS5kZFak8ppenCXB/SvXmSxwnDayEigRw/16tNVkndFFITUEgUhxNAVFHBDH5XNQqprdTEjoyHQ5LRRURHXa/HNN1wvBsDNobJsGTBs2Mv8GXV6LUJCuFNe7dtzw0r79NHdNlZUm4OblDE8KyuTByUpYjGelpYi4nkOLuTn1rptztnZ2Pbttxhx4UK1PR2vkpU/07MnjixbhkYNG6L1i8DDx9wcAiaAQABIof2De3UBPcANP37ypPq5dirSVZK3DAUhtURBCHndPHvGJSf27q3715o7F1i3rvpyhjK7a13Kz+eupbNmDZCTw+WI3L/P9RypyrEZM4Y7hSPL/8nMBMLDudEbNUn+rQ1tH9w+3JaNnS2r7115K9sTvdubIre8nLtJJC/vV3j89tmz2L5uHWyKimCsxmyyEqEQzNoabNs2mIwdW+n5hw+56+BMnMhd6E7b26/LgF5XSd6AhsdQpgc2b97MGjVqxEQiEevcuTO7fPlyleWPHDnCmjdvzkQiEWvdujU7deqUwvP5+flsxowZzNPTk5mZmbGWLVuybdu2qd2e3NxcBoDl5ubWaHsI0Se5uYy1b8+YiQljR4/q/vUGDGCMywap+hYRofu2GKrsbMYWL2Zs4ULu8bFjjAkEqt/LNm14ba6CYxkZrMHFiwwREfKb18WL7FhGhsZ1lYilTHTyIkN4hEJ98lt4BLM6dZGVS6Vq1VculbKcJ09YyltvMWk1/6BSgD0LDmYsPV1lfatXc8VtbBh7/Fi7219QwNiBA+p9lg4c0Khqhfcj4vlzdiAtjUU8f672+1gdTY6hvAchhw4dYqampuz7779ncXFx7IMPPmB2dnYsXcWOj4qKYkKhkK1atYrduXOHffHFF8zExITdunVLXuaDDz5gvr6+LCIigiUlJbEdO3YwoVDITp48qVabKAghrxOxmLExY7gvK6GQsR9+0G79ERGM3b378vGDB4xZW6s+aAoEjHl5MVZert12vK7Kyxlr0KDqg5CNDWOlpXy39CVtHtxm/ZjBcC6iciASHsFwLoKF/qR5cCP5z39YmVBY5ZtaJhQyyRdfVFlPeTljb7zBrdKnD2OyzazN9l+9ytjIkdw+/+MPwwzoDSoI6dy5M5sxY4b8sUQiYR4eHiwsLExp+dGjR7OBAwcqLAsMDGQffvih/LGfnx/76quvFMq0b9+e/ec//1GrTRSEkNdNeTljU6a8DAL++9/a1xkdzVjv3lydw4YpPif75f5qICJbduxY7V+/voiIMMwDkTbIA7BuGQyHFXsXcOgiQ/eMmgW0AQFq9YSwtm2rreruXcbMzLjVtm+v2XZKpYydPs1Yr16KzfjlF277DS2gN5ggpLS0lAmFQnbixAmF5ZMmTWKDBw9Wuo6Xlxf79ttvFZYtWbKEtanQH/nBBx+wjh07sidPnjCpVMrOnTvHrKys2F9//aW0zpKSEpabmyu/JScnUxBCXjsSCWMff/zyC2zTpprVc+MGY++887IeExPGZsyo/EV47FjlX/BeXhSAaErXXfL6TCEAM5IyBDxneCuN+2skrVkAlppa6c2TvDjKS5Qd7dPSqq3y22+5opaWXE+gusrKGNu/n7GAgJcvZ2zM2MSJjP3zD1fGEAN6gwlCUlJSGAB28eJFheXz589nnTt3VrqOiYkJO/DKp23Lli3MxcVF/rikpIRNmjSJAWDGxsbM1NSU7d27V2U7li5dygBUulEQQl43UiljoaEvv8g0SJVi8fGMjRr1cl0jI8amTmUsKUn1OuXl3AHiwAHur779YjME9bknRCcB2J49ij0eQiET29qyf0JDmdjWlklfPU1TxbFDRiJhrHt3rniPHtwpUHX+72NjX76MpSVjISGMPXpUuZyhBfSaBCG1nBJHP23atAmXLl3Czz//jEaNGuH8+fOYMWMGPDw8EBwcXKn8okWLEBoaKn+cl5cHLy+vumwyIXVCIOBGXlhYcBf4kg3fVGfo56lTwI8/cvfHjuWuAdO8edWvJxS+3sNw60K3btwomKquoNygAVfudePurt1yAIDffuOGDb04ngsGD4bJ9u3wd3EBFizg5rI/cYJ7YwUCrvykSVVWaWQE7N7NXe/m/n3uIpKpqS+fl80U3K0bEB0NDB7MLQ8IACZMAFq0AD75BFAySSsAbmK/IUO0dzFMvVIHQZFKujgdU1RUxExMTNivv/6qUOa9995j/fr1U6tdlBNC6oNnz7i/yn5lNWjA2M6djF279rJ8URGXV3LzJj/trc8MsUteG2Q5IVrLiSgr47J4Acbs7Bg7fFh5ucOHuecBrryaL7BihfK2ypaZmnK3p0/VbK+B0uQYWsejyBWZmpqiQ4cOCA8Ply+TSqUIDw9HUFCQ0nWCgoIUygPAmTNn5OXLyspQVlYGo1cGyAuFQkjVGBdOSH3h6PhyNsZXJ0N68gSYPp2bKEsi4ZaZm7/8tUfq1vDh3FT3np6Kyxs0eL2nwBcKuR4EoPLVcWWP16/XoEeguBho3Jj7x05IUH4pXIBbnpDAlfP15WaTq4ZEwk04p6y3SrZMLAZat+am6icv1EFQVKVDhw4xkUjE9uzZw+7cucOmT5/O7OzsWNqLZKCJEyeyhbLB8owbomtsbMzWrFnD4uPj2dKlSysN0e3Rowfz8/NjERER7MGDB2z37t3MzMyMbd26Va02UU8IqQ/UGfppaspYSgrfLSUy9TXHRqs5EZq+aWqWVzd359w5zZtsaAwqJ2TMmDHIzMzEkiVLkJaWhrZt2+L06dNwfXGBn8ePHyv0anTp0gUHDhzAF198gc8//xxNmzbFTz/9hNatW8vLHDp0CIsWLcKECRPw/PlzNGrUCF9//TU++uijOt8+QvTVhQtVTwcNcL/c/v0X8PComzaRqtXXHBut5kRoupKa5SvmgFRFjQvY1is0bbsSNG07qQ8OHgTGj6++HE2vTkj16JpJL2lyDOU1J4QQwh+djDwgpJ6SjWJSdW06gQDw8no9RzHVBgUhhNRT9KVJiPZoPYm2nqAghJB6ir40CdGu+jqKqTYoJ0QJygkh9YmyS8R7eXEBCH1pEqI5dSb/e51pcgylIEQJCkJIfVPfvzQJIdqjyTGU9yG6hBD+1dehn4QQflFOCCGEEEJ4QUEIIYQQQnhBQQghhBBCeEFBCCGEEEJ4QUEIIYQQQnhBQQghhBBCeEFDdJWQTZ2Sl5fHc0sIIYQQwyI7dqozDRkFIUpkZWUBALy8vHhuCSGEEGKY8vPzYWtrW2UZCkKUcHBwAAA8fvy42jfQUOTl5cHLywvJycmvzSywtE2G4XXbptdtewDaJkNhKNvEGEN+fj48PDyqLUtBiBJGRlyqjK2trV7v6JqwsbGhbTIAtE3673XbHoC2yVAYwjap+wOeElMJIYQQwgsKQgghhBDCCwpClBCJRFi6dClEIhHfTdEa2ibDQNuk/1637QFomwzF67hNAqbOGBpCCCGEEC2jnhBCCCGE8IKCEEIIIYTwgoIQQgghhPCCghBCCCGE8KLeBiFbtmyBt7c3zMzMEBgYiCtXrlRZ/scff0SLFi1gZmYGf39//Pbbb3XU0uqFhYWhU6dOsLa2houLC4YOHYqEhIQq19mzZw8EAoHCzczMrI5aXL0vv/yyUvtatGhR5Tr6vI8AwNvbu9I2CQQCzJgxQ2l5fdxH58+fx6BBg+Dh4QGBQICffvpJ4XnGGJYsWQJ3d3eYm5sjODgY9+7dq7ZeTT+P2lTVNpWVlWHBggXw9/eHpaUlPDw8MGnSJDx9+rTKOmvy/6st1e2jKVOmVGrb22+/XW29+rqPACj9XAkEAqxevVplnXzuI0C97+2SkhLMmDEDjo6OsLKywogRI5Cenl5lvTX9DPKlXgYhhw8fRmhoKJYuXYqYmBgEBASgX79+yMjIUFr+4sWLGDduHN577z3cuHEDQ4cOxdChQ3H79u06brlyf/31F2bMmIFLly7hzJkzKCsrQ9++fVFYWFjlejY2NkhNTZXfHj16VEctVo+fn59C+/7++2+VZfV9HwHA1atXFbbnzJkzAIBRo0apXEff9lFhYSECAgKwZcsWpc+vWrUKGzduxPbt23H58mVYWlqiX79+KCkpUVmnpp9Hbatqm4qKihATE4PFixcjJiYGx48fR0JCAgYPHlxtvZr8/2pTdfsIAN5++22Fth08eLDKOvV5HwFQ2JbU1FR8//33EAgEGDFiRJX18rWPAPW+t+fMmYNffvkFP/74I/766y88ffoUw4cPr7LemnwGecXqoc6dO7MZM2bIH0skEubh4cHCwsKUlh89ejQbOHCgwrLAwED24Ycf6rSdNZWRkcEAsL/++ktlmd27dzNbW9u6a5SGli5dygICAtQub2j7iDHGZs+ezXx9fZlUKlX6vL7vIwDsxIkT8sdSqZS5ubmx1atXy5fl5OQwkUjEDh48qLIeTT+PuvTqNilz5coVBoA9evRIZRlN/391Rdn2TJ48mQ0ZMkSjegxtHw0ZMoS99dZbVZbRl30k8+r3dk5ODjMxMWE//vijvEx8fDwDwKKjo5XWUdPPIJ/qXU+IWCzG9evXERwcLF9mZGSE4OBgREdHK10nOjpaoTwA9OvXT2V5vuXm5gJ4eSE+VQoKCtCoUSN4eXlhyJAhiIuLq4vmqe3evXvw8PBA48aNMWHCBDx+/FhlWUPbR2KxGD/88AOmTZsGgUCgspy+76OKkpKSkJaWprAfbG1tERgYqHI/1OTzyLfc3FwIBALY2dlVWU6T/9+6FhkZCRcXFzRv3hwff/yx/MrhyhjaPkpPT8epU6fw3nvvVVtWn/bRq9/b169fR1lZmcL73qJFCzRs2FDl+16TzyDf6l0Q8uzZM0gkEri6uiosd3V1RVpamtJ10tLSNCrPJ6lUipCQEHTt2hWtW7dWWa558+b4/vvvcfLkSfzwww+QSqXo0qULnjx5UoetVS0wMBB79uzB6dOnsW3bNiQlJaFbt27Iz89XWt6Q9hEA/PTTT8jJycGUKVNUltH3ffQq2XutyX6oyeeRTyUlJViwYAHGjRtX5QXENP3/rUtvv/029u3bh/DwcHzzzTf466+/0L9/f0gkEqXlDW0f7d27F9bW1tWettCnfaTsezstLQ2mpqaVgt3qjlWyMuquwze6iu5rZsaMGbh9+3a15zaDgoIQFBQkf9ylSxe0bNkSO3bswPLly3XdzGr1799ffr9NmzYIDAxEo0aNcOTIEbV+4ei77777Dv3796/yUtf6vo/qm7KyMowePRqMMWzbtq3Ksvr8/zt27Fj5fX9/f7Rp0wa+vr6IjIxE7969eWyZdnz//feYMGFCtUnc+rSP1P3efh3Vu54QJycnCIXCShnG6enpcHNzU7qOm5ubRuX5MnPmTPz666+IiIhAgwYNNFrXxMQE7dq1Q2Jioo5aVzt2dnZo1qyZyvYZyj4CgEePHuHs2bN4//33NVpP3/eR7L3WZD/U5PPIB1kA8ujRI5w5c0bjy6hX9//Lp8aNG8PJyUll2wxlHwHAhQsXkJCQoPFnC+BvH6n63nZzc4NYLEZOTo5C+eqOVbIy6q7Dt3oXhJiamqJDhw4IDw+XL5NKpQgPD1f41VlRUFCQQnkAOHPmjMrydY0xhpkzZ+LEiRM4d+4cfHx8NK5DIpHg1q1bcHd310ELa6+goAD3799X2T5930cV7d69Gy4uLhg4cKBG6+n7PvLx8YGbm5vCfsjLy8Ply5dV7oeafB7rmiwAuXfvHs6ePQtHR0eN66ju/5dPT548QVZWlsq2GcI+kvnuu+/QoUMHBAQEaLxuXe+j6r63O3ToABMTE4X3PSEhAY8fP1b5vtfkM8g7nhNjeXHo0CEmEonYnj172J07d9j06dOZnZ0dS0tLY4wxNnHiRLZw4UJ5+aioKGZsbMzWrFnD4uPj2dKlS5mJiQm7desWX5ug4OOPP2a2trYsMjKSpaamym9FRUXyMq9u07Jly9gff/zB7t+/z65fv87Gjh3LzMzMWFxcHB+bUMncuXNZZGQkS0pKYlFRUSw4OJg5OTmxjIwMxpjh7SMZiUTCGjZsyBYsWFDpOUPYR/n5+ezGjRvsxo0bDABbt24du3HjhnykyMqVK5mdnR07efIk++eff9iQIUOYj48PKy4ultfx1ltvsU2bNskfV/d55HObxGIxGzx4MGvQoAGLjY1V+HyVlpaq3Kbq/n/52p78/Hw2b948Fh0dzZKSktjZs2dZ+/btWdOmTVlJSYnK7dHnfSSTm5vLLCws2LZt25TWoU/7iDH1vrc/+ugj1rBhQ3bu3Dl27do1FhQUxIKCghTqad68OTt+/Lj8sTqfQX1SL4MQxhjbtGkTa9iwITM1NWWdO3dmly5dkj/Xo0cPNnnyZIXyR44cYc2aNWOmpqbMz8+PnTp1qo5brBoApbfdu3fLy7y6TSEhIfLtd3V1ZQMGDGAxMTF133gVxowZw9zd3ZmpqSnz9PRkY8aMYYmJifLnDW0fyfzxxx8MAEtISKj0nCHso4iICKX/a7J2S6VStnjxYubq6spEIhHr3bt3pW1t1KgRW7p0qcKyqj6PulbVNiUlJan8fEVERKjcpur+f/nanqKiIta3b1/m7OzMTExMWKNGjdgHH3xQKZgwpH0ks2PHDmZubs5ycnKU1qFP+4gx9b63i4uL2SeffMLs7e2ZhYUFGzZsGEtNTa1UT8V11PkM6hMBY4zppo+FEEIIIUS1epcTQgghhBD9QEEIIYQQQnhBQQghhBBCeEFBCCGEEEJ4QUEIIYQQQnhBQQghhBBCeEFBCCGEEEJ4QUEIIYQQQnhBQQghpF6IjIyEQCCodEEwQgh/KAghhBBCCC8oCCGEEEIILygIIYTUCalUirCwMPj4+MDc3BwBAQE4evQogJenSk6dOoU2bdrAzMwMb7zxBm7fvq1Qx7Fjx+Dn5weRSARvb2+sXbtW4fnS0lIsWLAAXl5eEIlEaNKkCb777juFMtevX0fHjh1hYWGBLl26ICEhQbcbTghRiYIQQkidCAsLw759+7B9+3bExcVhzpw5ePfdd/HXX3/Jy8yfPx9r167F1atX4ezsjEGDBqGsrAwAFzyMHj0aY8eOxa1bt/Dll19i8eLF2LNnj3z9SZMm4eDBg9i4cSPi4+OxY8cOWFlZKbTjP//5D9auXYtr167B2NgY06ZNq5PtJ4QowfdlfAkhr7+SkhJmYWHBLl68qLD8vffeY+PGjZNfqv3QoUPy57Kyspi5uTk7fPgwY4yx8ePHsz59+iisP3/+fNaqVSvGGGMJCQkMADtz5ozSNshe4+zZs/Jlp06dYgBYcXGxVraTEKIZ6gkhhOhcYmIiioqK0KdPH1hZWclv+/btw/379+XlgoKC5PcdHBzQvHlzxMfHAwDi4+PRtWtXhXq7du2Ke/fuQSKRIDY2FkKhED169KiyLW3atJHfd3d3BwBkZGTUehsJIZoz5rsBhJDXX0FBAQDg1KlT8PT0VHhOJBIpBCI1ZW5urlY5ExMT+X2BQACAy1chhNQ96gkhhOhcq1atIBKJ8PjxYzRp0kTh5uXlJS936dIl+f3s7Gz8+++/aNmyJQCgZcuWiIqKUqg3KioKzZo1g1AohL+/P6RSqUKOCSFEv1FPCCFE56ytrTFv3jzMmTMHUqkUb775JnJzcxEVFQUbGxs0atQIAPDVV1/B0dERrq6u+M9//gMnJycMHToUADB37lx06tQJy5cvx5gxYxAdHY3Nmzdj69atAABvb29MnjwZ06ZNw8aNGxEQEIBHjx4hIyMDo0eP5mvTCSFVoCCEEFInli9fDmdnZ4SFheHBgwews7ND+/bt8fnnn8tPh6xcuRKzZ8/GvXv30LZtW/zyyy8wNTUFALRv3x5HjhzBkiVLsHz5cri7u+Orr77ClClT5K+xbds2fP755/jkk0+QlZWFhg0b4vPPP+djcwkhahAwxhjfjSCE1G+RkZHo1asXsrOzYWdnx3dzCCF1hHJCCCGEEMILCkIIIYQQwgs6HUMIIYQQXlBPCCGEEEJ4QUEIIYQQQnhBQQghhBBCeEFBCCGEEEJ4QUEIIYQQQnhBQQghhBBCeEFBCCGEEEJ4QUEIIYQQQnjx/1Z/Sj3J0bNrAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='21' class='progress-bar-interrupted' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      42.00% [21/50 44:19<1:01:12][earlystopping]\n",
       "      <br>\n",
       "      \n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 3 epoch,early stopping >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.229796</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.179390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.180542</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.121662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.119084</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.106987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.103120</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.102721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.113289</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.093944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.091118</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.090922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.089273</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.091708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.088009</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.090138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.076904</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.094956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.082883</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.092965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.098641</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.087801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.089146</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.088401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.086661</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.085790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.080737</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.088450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.086944</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.088034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.093868</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.084467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.080087</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.083260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.094242</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.082587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>0.077127</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.084586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>0.082468</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.086839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>0.088173</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.084884</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss        lr  val_loss\n",
       "0       1    0.229796  0.000002  0.179390\n",
       "1       2    0.180542  0.000002  0.121662\n",
       "2       3    0.119084  0.000002  0.106987\n",
       "3       4    0.103120  0.000002  0.102721\n",
       "4       5    0.113289  0.000002  0.093944\n",
       "5       6    0.091118  0.000002  0.090922\n",
       "6       7    0.089273  0.000002  0.091708\n",
       "7       8    0.088009  0.000002  0.090138\n",
       "8       9    0.076904  0.000002  0.094956\n",
       "9      10    0.082883  0.000002  0.092965\n",
       "10     11    0.098641  0.000002  0.087801\n",
       "11     12    0.089146  0.000002  0.088401\n",
       "12     13    0.086661  0.000002  0.085790\n",
       "13     14    0.080737  0.000002  0.088450\n",
       "14     15    0.086944  0.000002  0.088034\n",
       "15     16    0.093868  0.000002  0.084467\n",
       "16     17    0.080087  0.000002  0.083260\n",
       "17     18    0.094242  0.000002  0.082587\n",
       "18     19    0.077127  0.000002  0.084586\n",
       "19     20    0.082468  0.000002  0.086839\n",
       "20     21    0.088173  0.000002  0.084884"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=50,patience=3,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path,\n",
    "                mixed_precision='fp16'\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 四，验证模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95bdaf75db1945a298ef81d3a1e91de4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from peft import PeftModel \n",
    "model = AutoModel.from_pretrained(\"chatglm2-6b\",\n",
    "                                  load_in_8bit=False, \n",
    "                                  trust_remote_code=True, \n",
    "                                  device_map='auto')\n",
    "model = PeftModel.from_pretrained(model,ckpt_path)\n",
    "model = model.merge_and_unload() #合并lora权重\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'好评'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "def predict(text):\n",
    "    response, history = model.chat(tokenizer, f\"{text} -> \", history=his,\n",
    "    temperature=0.01)\n",
    "    return response \n",
    "\n",
    "predict('死鬼，咋弄得这么有滋味呢') \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftest = pd.read_parquet('data/dftest.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = ['' for x in dftest['text']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [03:35<00:00,  9.26it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm \n",
    "for i in tqdm(range(len(dftest))):\n",
    "    text = dftest['text'].loc[i]\n",
    "    preds[i] = predict(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftest['pred'] = preds "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>pred</th>\n",
       "      <th>好评</th>\n",
       "      <th>差评</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tag</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>好评</th>\n",
       "      <td>888</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>差评</th>\n",
       "      <td>88</td>\n",
       "      <td>918</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "pred   好评   差评\n",
       "tag           \n",
       "好评    888  106\n",
       "差评     88  918"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.pivot_table(index='tag',columns = 'pred',values='text',aggfunc='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc= 0.903\n"
     ]
    }
   ],
   "source": [
    "acc = len(dftest.query('tag==pred'))/len(dftest)\n",
    "print('acc=',acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤗还行，用6000条数据，训练了一个小时左右，准确率到了90.3%，比未经微调的prompt方案的87.8%相比涨了两个多点~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以调整温度temperature参数，看看有没有机会把这个评论：\n",
    "\n",
    "'死鬼，咋弄得这么有滋味呢' 预测正确"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "差评\n",
      "好评\n",
      "好评\n",
      "好评\n",
      "差评\n",
      "差评\n",
      "好评\n",
      "差评\n",
      "差评\n",
      "好评\n"
     ]
    }
   ],
   "source": [
    "def predict(text,temperature=0.8):\n",
    "    response, history = model.chat(tokenizer, f\"{text} -> \", history=his,\n",
    "    temperature=temperature)\n",
    "    return response \n",
    "\n",
    "for i in range(10):\n",
    "    print(predict('死鬼，咋弄得这么有滋味呢')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，这个评论模型其实是不太吃得准它是好评还是差评的，毕竟，死鬼这个词的内涵太丰富了，跟字面的意思并不一样😋😋"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "如果在跑步比赛中超过了第二名,那么现在就是第二名。如果想要知道现在排名第几,需要知道自己和其他人的成绩。如果知道了所有人的成绩,就可以计算出自己在所有选手中的排名。\n"
     ]
    }
   ],
   "source": [
    "#模型的其它能力基本没有受到影响\n",
    "response, history = model.chat(tokenizer, \"跑步比赛如果你超过了第二名，你会成为第几名？\", history=[])\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 六，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以将模型和tokenizer都保存到一个新的路径，便于直接加载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_pretrained(\"chatglm2-6b-waimai\", max_shard_size='1GB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('chatglm2-6b-waimai/tokenizer_config.json',\n",
       " 'chatglm2-6b-waimai/special_tokens_map.json',\n",
       " 'chatglm2-6b-waimai/tokenizer.model',\n",
       " 'chatglm2-6b-waimai/added_tokens.json')"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.save_pretrained(\"chatglm2-6b-waimai\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还需要将相关的py文件也复制过去。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MODEL_LICENSE.txt\t\t  pytorch_model-00005-of-00007.bin\n",
      "README.md\t\t\t  pytorch_model-00006-of-00007.bin\n",
      "config.json\t\t\t  pytorch_model-00007-of-00007.bin\n",
      "configuration_chatglm.py\t  pytorch_model.bin.index.json\n",
      "modeling_chatglm.py\t\t  quantization.py\n",
      "pytorch_model-00001-of-00007.bin  tokenization_chatglm.py\n",
      "pytorch_model-00002-of-00007.bin  tokenizer.model\n",
      "pytorch_model-00003-of-00007.bin  tokenizer_config.json\n",
      "pytorch_model-00004-of-00007.bin\n"
     ]
    }
   ],
   "source": [
    "!ls chatglm2-6b  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp  chatglm2-6b/*.py chatglm2-6b-waimai/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "config.json\t\t\t  pytorch_model-00010-of-00015.bin\n",
      "configuration_chatglm.py\t  pytorch_model-00011-of-00015.bin\n",
      "generation_config.json\t\t  pytorch_model-00012-of-00015.bin\n",
      "modeling_chatglm.py\t\t  pytorch_model-00013-of-00015.bin\n",
      "pytorch_model-00001-of-00015.bin  pytorch_model-00014-of-00015.bin\n",
      "pytorch_model-00002-of-00015.bin  pytorch_model-00015-of-00015.bin\n",
      "pytorch_model-00003-of-00015.bin  pytorch_model.bin.index.json\n",
      "pytorch_model-00004-of-00015.bin  quantization.py\n",
      "pytorch_model-00005-of-00015.bin  special_tokens_map.json\n",
      "pytorch_model-00006-of-00015.bin  tokenization_chatglm.py\n",
      "pytorch_model-00007-of-00015.bin  tokenizer.model\n",
      "pytorch_model-00008-of-00015.bin  tokenizer_config.json\n",
      "pytorch_model-00009-of-00015.bin\n"
     ]
    }
   ],
   "source": [
    "!ls chatglm2-6b-waimai "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4266bf39e1ee4005800ddab98e190444",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/15 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import  AutoModel,AutoTokenizer\n",
    "model_name = \"chatglm2-6b-waimai\" \n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_name, trust_remote_code=True)\n",
    "model = AutoModel.from_pretrained(model_name,\n",
    "        trust_remote_code=True).half().cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\"\"文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\n",
    "\n",
    "下面是一些范例:\n",
    "\n",
    "味道真不错 -> 好评\n",
    "太辣了，吃不下都  -> 差评\n",
    "\n",
    "请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\n",
    "\n",
    "xxxxxx ->\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prompt.replace('xxxxxx',text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评\n"
     ]
    }
   ],
   "source": [
    "response, his = model.chat(tokenizer, get_prompt('狗子，怎么做的这么好吃呀？'), history=[])\n",
    "print(response)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "感觉不错的话，记得点个star支持一下一个有毅力的吃货哦！"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "25273a2a68c96ebac13d7fb9e0db516f9be0772777a0507fe06d682a441a3ba7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}