7-6,DeepCross模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "71bcfb14",
   "metadata": {},
   "source": [
    "\n",
    "# 7-6，DeepCross模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c91553e",
   "metadata": {},
   "source": [
    "谷歌在CTR预估和推荐排序模型方面有3篇相对重要的文章。\n",
    "\n",
    "第1篇是2016年的Deep&Wide,第2篇是2017年的Deep&Cross(DCN),第3篇是2020年的DCN-V2.\n",
    "\n",
    "这3篇文章的核心思想分别用1句话概括如下：\n",
    "\n",
    "第1篇Deep&Wide说，线性回归LR具有好的记忆能力(Wide部分),多层感知机DNN具有好的泛化能力(Deep部分),把它们并行的结合起来食用口感真香.\n",
    "\n",
    "第2篇Deep&Cross说，将Deep&Wide的Wide部分换成一个CrossNet，可以自动捕获多层级特征交叉，减少人工特征工程。新的模型叫做DCN口感更香！\n",
    "\n",
    "第3篇DCNV2说，DCN的CrossNet的仅仅用一个向量来建模交叉能力, 拟合能力太弱, 配方升级换成矩阵吧。升级后的DCNV2量大管饱，又香又饱！什么，矩阵太大跑的慢，不太好消化? 别急，我们还有秘制配方，把矩阵换成两个低秩矩阵的乘积，这个用了秘制配方的模型叫做DCN-Mix，嘎嘣脆，又香又饱还好消化，你值得拥有！\n",
    "\n",
    "今天我们就来尝尝DCN、DCNV2以及嘎嘣脆的DCN-Mix！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672ebc71",
   "metadata": {},
   "source": [
    "参考文档：\n",
    "\n",
    "* Deep&Wide论文： https://arxiv.org/pdf/1606.07792v1.pdf\n",
    "\n",
    "* DCN论文：https://arxiv.org/pdf/1708.05123.pdf \n",
    "\n",
    "* DCNV2论文： https://arxiv.org/pdf/2008.13535.pdf\n",
    "\n",
    "* 解密Deep&Cross: https://zhuanlan.zhihu.com/p/55234968 \n",
    "\n",
    "* 大规模排序系统中的特征交叉DCN-V2: https://zhuanlan.zhihu.com/p/353223660\n",
    "\n",
    "* 代码实现参考：https://github.com/shenweichen/DeepCTR-Torch/blob/bc881dcd417fec64f840b0cacce124bc86b3687c/deepctr_torch/layers/interaction.py#L406-L537\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39112727",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5eb6a1a",
   "metadata": {},
   "source": [
    "## 一，DeepCross原理解析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14735953",
   "metadata": {},
   "source": [
    "谷歌的DeepCross系列模型(DCN-vector, DCN-matrix, DCN-mix) 围绕解决的核心问题三高问题。\n",
    "\n",
    "如何实现 高效率，高表达能力 的 高阶显式特征交叉？\n",
    "\n",
    "这三高问题分别是什么含义呢？\n",
    "\n",
    "1，高阶：至少支持三阶特征交叉。实践证明，在CTR和推荐领域，二阶特征交叉和三阶特征交叉是有用的，四阶及以上的特征交叉没啥用。FM只是二阶显式特征交叉。 \n",
    "\n",
    "2，高效率：特征交叉模块的预测效率最好是O(n),其中n为特征数量。\n",
    "\n",
    "3，高表达能力：特征交叉模块有足够大的参数容量来捕获各种可能的特征交叉。\n",
    "\n",
    "下面我们分别看看 DCN-vector， DCN-matrix, DCN-mix在三高问题的解决思路和解决程度。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b5aa021",
   "metadata": {},
   "source": [
    "### 1， DCN-vector "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5959a9",
   "metadata": {},
   "source": [
    "DCN-vector 也就是 2017年发布的最初版本的DCN， 很好地解决了三高问题中的两高。\n",
    "\n",
    "1，高阶 (DCN-vector可以很方便地支持三阶及以上的显式特征交叉)\n",
    "\n",
    "2，高效率 (DCN-vector特征交叉模块的推理复杂度是O(n)，非常高效，和FM不分伯仲)\n",
    "\n",
    "我们来看看DCN-vector怎么做的显式特征交叉。\n",
    "\n",
    "先看图和公式。\n",
    "\n",
    "有两个要点，一个是用和x0等长的参数向量来捕获交叉关系，第二个是参照了残差模块的设计将低阶量加回到输出中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a7563e",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wduan1hmj20a30570sv.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b78c0cf",
   "metadata": {},
   "source": [
    "代码和图一样好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e109057",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetVector(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,1,bias=False) for i in range(self.n_cross)])\n",
    "        self.biases = nn.ParameterList(\n",
    "            [nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+self.biases[i]+xi\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fcd0b6",
   "metadata": {},
   "source": [
    "CrossNetVector的结构非常简洁。每升高一阶特征交叉，主要运算是将上一阶特征向量xi和交叉模块参数向量w做点积，并数乘原始特征向量x0。\n",
    "\n",
    "该计算的计算前后向量的维度不变，计算量和特征向量的长度n成正比，所以复杂度是O(n).\n",
    "\n",
    "值得注意的是，虽然CrossNetVector的复杂度很低仅仅是O(n),  但是 $x_{0}*x^{'}*\\omega$ 这一项展开后是包括任意两个特征的交叉乘积贡献的。\n",
    "\n",
    "我们知道任意两个特征的交叉共有 n(n-1)/2 种组合方式，这个复杂度是O(n^2)的，那么为什么 CrossNetVector 能够以 O(n)的复杂度实现包括O(n^2)量级的特征交叉组合贡献的呢？\n",
    "\n",
    "原因是CrossNetVector这种结构实际上具有权重共享的特性。\n",
    "\n",
    "假设有5个特征 x1,x2,x3,x4,x5, 如果要独立地描述它们之间的两两交叉要10个参数，但是CrossNetVector中仅仅有5个表述特征交叉系数的参数 w1,w2,w3,w4,w5\n",
    "\n",
    "其中 x1x2的交叉项的系数是 (w1+w2), x1x3的交叉项系数时 (w1+w3),...,x1x5的交叉项系数是 (w1+w5), 可见，w1这个参数被用在了 x1和任何参数的交叉项中作为组成部分。\n",
    "\n",
    "这就是一种权重共享的特性，这种特性有助于DCN变得轻便高效，并像FM那样学习到稀疏特征的交叉。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4965699",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4daccd07",
   "metadata": {},
   "source": [
    "### 2, DCN-matrix "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8114c672",
   "metadata": {},
   "source": [
    "DCN-vector 虽然解决了三高中的两高: 高阶、高效率。\n",
    "\n",
    "但是它在另一高也就是高表达能力方面有些缺陷，它仅仅用一个和输入特征等长的向量来刻画特征之间的相互作用，不足以表述丰富的特征交叉模式。\n",
    "\n",
    "CrossNetVector的参数数量相比Deep部分的MLP的参数数量来看实在微不足道，可能存在记忆容量瓶颈。\n",
    "\n",
    "换言之，它的权值共享可能做的有些过头了。\n",
    "\n",
    "正是基于这样的洞察，DCN-matrix 在DCN-vector基础上做了一些轻微的改动，用矩阵代替了向量，并适配性地修改了一些运算符，强化了显式特征交叉项的表达能力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76da09f6",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wgsq1zcyj208u0473yj.jpg) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bce4a3d7",
   "metadata": {},
   "source": [
    "代码比图更加好懂。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46e93a0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetMatrix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,d_in) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+xi\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "337a9a52",
   "metadata": {},
   "source": [
    "可以看到 CrossNetMatrix 用一个 $n\\times n$ 的交互矩阵$W$来刻画 特征之间的交互，实际上$W$的第i行第j列的元素 $W_{ij}$\n",
    "\n",
    "表达的就是第i个特征和第j个特征之间的交互项系数。 因此 CrossNetMatrix 的复杂度是O(n^2)的，并且具有很好的解释型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b88a0f8",
   "metadata": {},
   "source": [
    "于是 CrossNetMatrix整体来说是牺牲了 高效性，换来了高表达能力。不过，在许多特征数量不算特别多的情况下，例如n在1000以下的时候，CrossNetMatrix依旧是非常高效的。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e28fe65",
   "metadata": {},
   "source": [
    "### 3, DCN-mix "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a18321",
   "metadata": {},
   "source": [
    "DCN-mix可以说是 DCN-matrix基础上提升计算效率，同时保持高表达能力的一种方案。 \n",
    "\n",
    "其主要思想洞察如下。\n",
    "\n",
    "1，矩阵分解：根据SVD主成分分析理论，庞大的交互矩阵W可以由为两个低秩矩阵U和V的乘积来近似。这样可以将参数量和计算复杂度降低到O(n)。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wid7h72rj209s01ga9v.jpg)\n",
    "\n",
    "2，专家融合：使用两个低秩矩阵U和V的乘积来近似W提升了效率但是降低了表达能力，可以使用MOE思想(Mixture of Experts)融合多个低秩空间的专家网络来提升表达能力。\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wibupquej20eb0ayjrt.jpg)\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wiet82zxj209302ma9y.jpg)\n",
    "\n",
    "3，低秩变换：可以在U和V之间在低秩空间上做一些变换适当再拉一拉表达能力。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wifm6877j20ac01d744.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fbbc1e",
   "metadata": {},
   "source": [
    "整体来说，DCN-mix用到的这3个技巧还是有些难度的，可以结合代码理解一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0db9756",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "class CrossNetMix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross =2, low_rank=32, n_experts=4):\n",
    "        super().__init__()\n",
    "        self.d_in = d_in\n",
    "        self.n_cross = n_cross\n",
    "        self.low_rank = low_rank\n",
    "        self.n_experts = n_experts\n",
    "\n",
    "        # U: (d_in, low_rank)\n",
    "        self.U_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # V: (d_in, low_rank)\n",
    "        self.V_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # C: (low_rank, low_rank)\n",
    "        self.C_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, low_rank, low_rank))) for i in range(self.n_cross)])\n",
    "\n",
    "        # G: (d_in, 1)\n",
    "        self.gating = nn.ModuleList([nn.Linear(d_in, 1, bias=False) for i in range(self.n_experts)])\n",
    "\n",
    "        # Bias \n",
    "        self.biases = nn.ParameterList([nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            output_of_experts = []\n",
    "            gating_score_of_experts = []\n",
    "            for expert_id in range(self.n_experts):\n",
    "\n",
    "                # (1) G(xi)\n",
    "                # compute the gating score by xi\n",
    "                gating_score_of_experts.append(self.gating[expert_id](xi))\n",
    "\n",
    "                # (2) E(xi)\n",
    "                # project the input xi to low_rank space\n",
    "                v_x = xi@(self.V_list[i][expert_id])   # (batch_size, low_rank)\n",
    "\n",
    "                # nonlinear activation in low rank space\n",
    "                v_x = torch.tanh(v_x)\n",
    "                v_x = v_x@self.C_list[i][expert_id]     # (batch_size, low_rank)\n",
    "                v_x = torch.tanh(v_x)\n",
    "\n",
    "                # project back to d_in space\n",
    "                uv_x = v_x@(self.U_list[i][expert_id].T)  # (batch_size, d_in)\n",
    "                expert_out = x0*(uv_x + self.biases[i])\n",
    "                output_of_experts.append(expert_out)\n",
    "\n",
    "            # (3) mixture of low-rank experts\n",
    "            output_of_experts = torch.stack(output_of_experts, 2)  # (batch_size, d_in, n_experts)\n",
    "            gating_score_of_experts = torch.stack(gating_score_of_experts, 1)  # (batch_size, n_experts, 1)\n",
    "            moe_out = torch.bmm(output_of_experts, gating_score_of_experts.softmax(1))\n",
    "            xi = torch.squeeze(moe_out) + xi  # (batch_size, d_in)\n",
    "\n",
    "        return xi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ac338ea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc2fcc0e",
   "metadata": {},
   "source": [
    "## 二，DeepCross的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbff274e",
   "metadata": {},
   "source": [
    "下面是DeepCross的一个pytorch完整实现。\n",
    "\n",
    "可以通过cross_type 来指定是哪个版本的DCN模型。\n",
    "\n",
    "* cross_type = \"vector\" 选择 DCNV1, 也可以称之为 DCN-vector \n",
    "\n",
    "* cross_type = \"matrix\" 选择 DCNV2, 也可以称之为 DCN-matrix \n",
    "\n",
    "* cross_type = \"mix\" 选择 DCN-mix \n",
    "\n",
    "除了三种CrossNet结构的实现外，还有两点细节值得注意：\n",
    "\n",
    "1，DeepCross对类别特征的处理与DeepFM等模型不太一样，是全部转换成embedding后再和数值类特征拼在一起，这种特性可以让各个类别特征根据其类别数量选择适当的embedding长度。\n",
    "\n",
    "2，DeepCross中Deep部分和Cross部分的关系可以是串行的(stacked=True)，也可以是并行的(stacked=False)。具体哪种效果更好，取决于数据集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c607ceba",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn,Tensor \n",
    "import torch.nn.functional as F \n",
    "\n",
    "#离散特征编码\n",
    "class CatEmbeddingSqrt(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征使用Embedding层编码, d_embed等于sqrt(category)\n",
    "    输入shape: [batch_size,d_in], \n",
    "    输出shape: [batch_size,d_out]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_embed_max = 100):\n",
    "        super().__init__()\n",
    "        self.categories = categories\n",
    "        self.d_embed_list = [min(max(int(x**0.5), 2),d_embed_max) for x in categories]\n",
    "        self.embedding_list = nn.ModuleList([nn.Embedding(self.categories[i],self.d_embed_list[i])\n",
    "                            for i in range(len(categories))])\n",
    "        self.d_cat_sum = sum(self.d_embed_list)\n",
    "        \n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        param x_cat: Long tensor of size ``(batch_size, d_in)``\n",
    "        \"\"\"\n",
    "        x_out = torch.cat([self.embedding_list[i](x_cat[:,i]) \n",
    "                           for i in range(len(self.categories)) ],dim=1)\n",
    "        return x_out\n",
    "    \n",
    "#deep部分\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, d_in, d_layers, dropout):\n",
    "        super().__init__()\n",
    "        layers = []\n",
    "        for d in d_layers:\n",
    "            layers.append(nn.Linear(d_in, d))\n",
    "            layers.append(nn.BatchNorm1d(d))\n",
    "            layers.append(nn.ReLU())\n",
    "            layers.append(nn.Dropout(p=dropout))\n",
    "            d_in = d\n",
    "        self.mlp = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.mlp(x)\n",
    "    \n",
    "\n",
    "#3种CrossNet的实现    \n",
    "class CrossNetVector(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,1,bias=False) for i in range(self.n_cross)])\n",
    "        self.biases = nn.ParameterList(\n",
    "            [nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+self.biases[i]+xi\n",
    "        return xi\n",
    "    \n",
    "    \n",
    "class CrossNetMatrix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross=2):\n",
    "        super().__init__()\n",
    "        self.n_cross = n_cross\n",
    "        self.linears = nn.ModuleList([nn.Linear(d_in,d_in) for i in range(self.n_cross)])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            xi = x0*self.linears[i](xi)+xi\n",
    "        return xi\n",
    "    \n",
    "\n",
    "class CrossNetMix(nn.Module):\n",
    "    def __init__(self, d_in, n_cross =2, low_rank=32, n_experts=4):\n",
    "        super().__init__()\n",
    "        self.d_in = d_in\n",
    "        self.n_cross = n_cross\n",
    "        self.low_rank = low_rank\n",
    "        self.n_experts = n_experts\n",
    "\n",
    "        # U: (d_in, low_rank)\n",
    "        self.U_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # V: (d_in, low_rank)\n",
    "        self.V_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, d_in, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # C: (low_rank, low_rank)\n",
    "        self.C_list = nn.ParameterList([nn.Parameter(nn.init.xavier_normal_(\n",
    "            torch.empty(n_experts, low_rank, low_rank))) for i in range(self.n_cross)])\n",
    "        \n",
    "        # G: (d_in, 1)\n",
    "        self.gating = nn.ModuleList([nn.Linear(d_in, 1, bias=False) for i in range(self.n_experts)])\n",
    "\n",
    "        # Bias \n",
    "        self.biases = nn.ParameterList([nn.Parameter(torch.zeros(d_in)) for i in range(self.n_cross)])\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        x0 = x\n",
    "        xi = x\n",
    "        for i in range(self.n_cross):\n",
    "            output_of_experts = []\n",
    "            gating_score_of_experts = []\n",
    "            for expert_id in range(self.n_experts):\n",
    "                \n",
    "                # (1) G(xi)\n",
    "                # compute the gating score by xi\n",
    "                gating_score_of_experts.append(self.gating[expert_id](xi))\n",
    "\n",
    "                # (2) E(xi)\n",
    "                # project the input xi to low_rank space\n",
    "                v_x = xi@(self.V_list[i][expert_id])   # (batch_size, low_rank)\n",
    "\n",
    "                # nonlinear activation in low rank space\n",
    "                v_x = torch.tanh(v_x)\n",
    "                v_x = v_x@self.C_list[i][expert_id]     # (batch_size, low_rank)\n",
    "                v_x = torch.tanh(v_x)\n",
    "\n",
    "                # project back to d_in space\n",
    "                uv_x = v_x@(self.U_list[i][expert_id].T)  # (batch_size, d_in)\n",
    "                expert_out = x0*(uv_x + self.biases[i])\n",
    "                output_of_experts.append(expert_out)\n",
    "\n",
    "            # (3) mixture of low-rank experts\n",
    "            output_of_experts = torch.stack(output_of_experts, 2)  # (batch_size, d_in, n_experts)\n",
    "            gating_score_of_experts = torch.stack(gating_score_of_experts, 1)  # (batch_size, n_experts, 1)\n",
    "            moe_out = torch.bmm(output_of_experts, gating_score_of_experts.softmax(1))\n",
    "            xi = torch.squeeze(moe_out) + xi  # (batch_size, d_in)\n",
    "            \n",
    "        return xi\n",
    "    \n",
    "    \n",
    "class DeepCross(nn.Module):\n",
    "    \"\"\"\n",
    "    DeepCross三种模型(DCN-vec,DCN-matrix,DCN-mix)的统一实现。\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, d_numerical, categories, d_embed_max = 8,\n",
    "                 n_cross=2, cross_type = \"matrix\", low_rank=32, n_experts=4, \n",
    "                 mlp_layers = [128,64,32] ,mlp_dropout = 0.25, \n",
    "                 stacked = True, \n",
    "                 n_classes = 1):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        if cross_type=='mix':\n",
    "            assert low_rank is not None and n_experts is not None\n",
    "        \n",
    "        if d_numerical is None:\n",
    "            d_numerical = 0\n",
    "        if categories is None:\n",
    "            categories = []\n",
    "            \n",
    "        self.categories = categories\n",
    "        self.n_classes = n_classes\n",
    "        self.stacked = stacked\n",
    "        \n",
    "        \n",
    "        self.cat_embedding = CatEmbeddingSqrt(categories, d_embed_max) if categories else None\n",
    "        \n",
    "        self.d_in = d_numerical \n",
    "        if self.cat_embedding:\n",
    "            self.d_in+=self.cat_embedding.d_cat_sum\n",
    "            \n",
    "        if cross_type==\"vector\":\n",
    "            self.cross_layer = CrossNetVector(self.d_in,n_cross)\n",
    "        elif cross_type==\"matrix\":\n",
    "            self.cross_layer = CrossNetMatrix(self.d_in,n_cross)\n",
    "        elif cross_type==\"mix\":\n",
    "            self.cross_layer = CrossNetMix(self.d_in,n_cross,low_rank,n_experts)\n",
    "        else:\n",
    "            raise NotImplementedError(\"cross_type should  be one of ('vector','matrix','mix') !\")\n",
    "        \n",
    "        self.mlp = MLP(\n",
    "            d_in= self.d_in,\n",
    "            d_layers = mlp_layers,\n",
    "            dropout = mlp_dropout\n",
    "        )\n",
    "        \n",
    "        if self.stacked:\n",
    "            self.last_linear = nn.Linear(mlp_layers[-1],n_classes)\n",
    "        else:\n",
    "            self.last_linear = nn.Linear(self.d_in+mlp_layers[-1],n_classes)\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        \"\"\"\n",
    "        x_num: numerical features\n",
    "        x_cat: category features\n",
    "        \"\"\"\n",
    "        x_num,x_cat = x\n",
    "        \n",
    "        #embedding \n",
    "        x_total = []\n",
    "        if x_num is not None:\n",
    "            x_total.append(x_num)\n",
    "        if self.cat_embedding is not None:\n",
    "            x_total.append(self.cat_embedding(x_cat))\n",
    "        x_total = torch.cat(x_total, dim=-1)\n",
    "        \n",
    "        \n",
    "        #cross部分\n",
    "        x_cross = self.cross_layer(x_total)\n",
    "        \n",
    "        \n",
    "        #deep部分\n",
    "        if self.stacked:\n",
    "            x_deep = self.mlp(x_cross)\n",
    "            x_out = self.last_linear(x_deep)\n",
    "        else:\n",
    "            x_deep = self.mlp(x_total)\n",
    "            x_deep_cross = torch.cat([x_deep,x_cross],axis = 1)\n",
    "            x_out = self.last_linear(x_deep_cross)\n",
    "            \n",
    "        if self.n_classes==1:\n",
    "            x_out = x_out.squeeze(-1)\n",
    "        \n",
    "        return x_out "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fb91e6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "##测试 DeepCross\n",
    "\n",
    "x_num = torch.randn(2,3)\n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "\n",
    "dcn_vec = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"vector\", \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = False,\n",
    "        n_classes = 1)\n",
    "\n",
    "dcn_matrix = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"matrix\", \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = True,\n",
    "        n_classes = 1)\n",
    "\n",
    "dcn_mix = DeepCross(d_numerical = 3, categories = [4,3,2], d_embed_max = 4,\n",
    "        n_cross=2, cross_type = \"mix\", low_rank=32, n_experts=4, \n",
    "        mlp_layers = [20,20], mlp_dropout=0.25,\n",
    "        stacked = False,\n",
    "        n_classes = 1)\n",
    "\n",
    "\n",
    "print(dcn_vec((x_num,x_cat)))\n",
    "print(dcn_matrix((x_num,x_cat)))\n",
    "print(dcn_mix((x_num,x_cat)))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516b5f6c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "371196d6",
   "metadata": {},
   "source": [
    "## 三，Criteo数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "888bd54a",
   "metadata": {},
   "source": [
    "Criteo数据集是一个经典的广告点击率CTR预测数据集。\n",
    "\n",
    "这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。\n",
    "\n",
    "数据集有13维数值特征(I1-I13)和26维类别特征(C14-C39), 共39维特征, 特征中包含着许多缺失值。\n",
    "\n",
    "训练集4000万个样本，测试集600万个样本。数据集大小超过100G.\n",
    "\n",
    "此处使用的是采样100万个样本后的cretio_small数据集。 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1c74075",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d6b5ddc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import datetime \n",
    "\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "import torch \n",
    "from torch import nn \n",
    "from torch.utils.data import Dataset,DataLoader  \n",
    "import torch.nn.functional as F \n",
    "import torchkeras \n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(info+'...\\n\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c79eaa4",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c810d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "\n",
    "dfdata = pd.read_csv(\"./eat_pytorch_datasets/criteo_small.zip\",sep=\"\\t\",header=None)\n",
    "dfdata.columns = [\"label\"] + [\"I\"+str(x) for x in range(1,14)] + [\n",
    "    \"C\"+str(x) for x in range(14,40)]\n",
    "\n",
    "cat_cols = [x for x in dfdata.columns if x.startswith('C')]\n",
    "num_cols = [x for x in dfdata.columns if x.startswith('I')]\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "for col in cat_cols:\n",
    "    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])\n",
    "\n",
    "dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])\n",
    "\n",
    "categories = [dfdata[col].max()+1 for col in cat_cols]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21d7ef14",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "#DataFrame转换成torch数据集Dataset, 特征分割成X_num,X_cat方式\n",
    "class DfDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 label_col,\n",
    "                 num_features,\n",
    "                 cat_features,\n",
    "                 categories,\n",
    "                 is_training=True):\n",
    "        \n",
    "        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None\n",
    "        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None\n",
    "        self.Y = torch.tensor(df[label_col].values).float() \n",
    "        self.categories = categories\n",
    "        self.is_training = is_training\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.Y)\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        if self.is_training:\n",
    "            return ((self.X_num[index],self.X_cat[index]),self.Y[index])\n",
    "        else:\n",
    "            return (self.X_num[index],self.X_cat[index])\n",
    "    \n",
    "    def get_categories(self):\n",
    "        return self.categories\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc41f88f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)\n",
    "dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)\n",
    "\n",
    "ds_train = DfDataset(dftrain,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_val = DfDataset(dfval,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_test = DfDataset(dftest,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51a784c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)\n",
    "dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)\n",
    "\n",
    "for features,labels in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937bec29",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63435a35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 此处我们选择的交叉模块是CrossNetMatrix, 也就是构建的DCNV2模型 \n",
    "# 读者也可以尝试CrossNetVector和CrossNetMix \n",
    "def create_net():\n",
    "    net = DeepCross(\n",
    "        d_numerical= ds_train.X_num.shape[1],\n",
    "        categories= ds_train.get_categories(),\n",
    "        d_embed_max = 8, \n",
    "        n_cross = 2, cross_type = \"matrix\",\n",
    "        mlp_layers = [128,64,32], mlp_dropout=0.25,\n",
    "        stacked = True,\n",
    "        n_classes = 1\n",
    "        \n",
    "    )\n",
    "    return net \n",
    "\n",
    "from torchkeras import summary\n",
    "\n",
    "net = create_net()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2869f709",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "33b34700",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a422cebd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys,time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime \n",
    "from tqdm import tqdm \n",
    "\n",
    "import torch\n",
    "from torch import nn \n",
    "from accelerate import Accelerator\n",
    "from copy import deepcopy\n",
    "\n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(str(info)+\"\\n\")\n",
    "    \n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn,stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None,\n",
    "                 accelerator = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator\n",
    "    \n",
    "    def __call__(self, features, labels):\n",
    "        #loss\n",
    "        preds = self.net(features)\n",
    "        loss = self.loss_fn(preds,labels)\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            if self.accelerator is  None:\n",
    "                loss.backward()\n",
    "            else:\n",
    "                self.accelerator.backward(loss)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        #metrics\n",
    "        step_metrics = {self.stage+\"_\"+name:metric_fn(preds, labels).item() \n",
    "                        for name,metric_fn in self.metrics_dict.items()}\n",
    "        return loss.item(),step_metrics\n",
    "    \n",
    "    \n",
    "class EpochRunner:\n",
    "    def __init__(self,steprunner):\n",
    "        self.steprunner = steprunner\n",
    "        self.stage = steprunner.stage\n",
    "        self.steprunner.net.train() if self.stage==\"train\" else self.steprunner.net.eval()\n",
    "        \n",
    "    def __call__(self,dataloader):\n",
    "        total_loss,step = 0,0\n",
    "        loop = tqdm(enumerate(dataloader), total =len(dataloader))\n",
    "        for i, batch in loop:\n",
    "            features,labels = batch\n",
    "            if self.stage==\"train\":\n",
    "                loss, step_metrics = self.steprunner(features,labels)\n",
    "            else:\n",
    "                with torch.no_grad():\n",
    "                    loss, step_metrics = self.steprunner(features,labels)\n",
    "\n",
    "            step_log = dict({self.stage+\"_loss\":loss},**step_metrics)\n",
    "\n",
    "            total_loss += loss\n",
    "            step+=1\n",
    "            if i!=len(dataloader)-1:\n",
    "                loop.set_postfix(**step_log)\n",
    "            else:\n",
    "                epoch_loss = total_loss/step\n",
    "                epoch_metrics = {self.stage+\"_\"+name:metric_fn.compute().item() \n",
    "                                 for name,metric_fn in self.steprunner.metrics_dict.items()}\n",
    "                epoch_log = dict({self.stage+\"_loss\":epoch_loss},**epoch_metrics)\n",
    "                loop.set_postfix(**epoch_log)\n",
    "\n",
    "                for name,metric_fn in self.steprunner.metrics_dict.items():\n",
    "                    metric_fn.reset()\n",
    "        return epoch_log\n",
    "\n",
    "class KerasModel(torch.nn.Module):\n",
    "    def __init__(self,net,loss_fn,metrics_dict=None,optimizer=None,lr_scheduler = None):\n",
    "        super().__init__()\n",
    "        self.accelerator = Accelerator()\n",
    "        self.history = {}\n",
    "        \n",
    "        self.net = net\n",
    "        self.loss_fn = loss_fn\n",
    "        self.metrics_dict = nn.ModuleDict(metrics_dict) \n",
    "        \n",
    "        self.optimizer = optimizer if optimizer is not None else torch.optim.Adam(\n",
    "            self.parameters(), lr=1e-2)\n",
    "        self.lr_scheduler = lr_scheduler\n",
    "        \n",
    "        self.net,self.loss_fn,self.metrics_dict,self.optimizer = self.accelerator.prepare(\n",
    "            self.net,self.loss_fn,self.metrics_dict,self.optimizer)\n",
    "\n",
    "    def forward(self, x):\n",
    "        if self.net:\n",
    "            return self.net.forward(x)\n",
    "        else:\n",
    "            raise NotImplementedError\n",
    "\n",
    "\n",
    "    def fit(self, train_data, val_data=None, epochs=10, ckpt_path='checkpoint.pt', \n",
    "            patience=5, monitor=\"val_loss\", mode=\"min\"):\n",
    "        \n",
    "        train_data = self.accelerator.prepare(train_data)\n",
    "        val_data = self.accelerator.prepare(val_data) if val_data else []\n",
    "\n",
    "        for epoch in range(1, epochs+1):\n",
    "            printlog(\"Epoch {0} / {1}\".format(epoch, epochs))\n",
    "            \n",
    "            # 1，train -------------------------------------------------  \n",
    "            train_step_runner = StepRunner(net = self.net,stage=\"train\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    optimizer = self.optimizer, lr_scheduler = self.lr_scheduler,\n",
    "                    accelerator = self.accelerator)\n",
    "            train_epoch_runner = EpochRunner(train_step_runner)\n",
    "            train_metrics = train_epoch_runner(train_data)\n",
    "            \n",
    "            for name, metric in train_metrics.items():\n",
    "                self.history[name] = self.history.get(name, []) + [metric]\n",
    "\n",
    "            # 2，validate -------------------------------------------------\n",
    "            if val_data:\n",
    "                val_step_runner = StepRunner(net = self.net,stage=\"val\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    accelerator = self.accelerator)\n",
    "                val_epoch_runner = EpochRunner(val_step_runner)\n",
    "                with torch.no_grad():\n",
    "                    val_metrics = val_epoch_runner(val_data)\n",
    "                val_metrics[\"epoch\"] = epoch\n",
    "                for name, metric in val_metrics.items():\n",
    "                    self.history[name] = self.history.get(name, []) + [metric]\n",
    "            \n",
    "            # 3，early-stopping -------------------------------------------------\n",
    "            arr_scores = self.history[monitor]\n",
    "            best_score_idx = np.argmax(arr_scores) if mode==\"max\" else np.argmin(arr_scores)\n",
    "            if best_score_idx==len(arr_scores)-1:\n",
    "                torch.save(self.net.state_dict(),ckpt_path)\n",
    "                print(\"<<<<<< reach best {0} : {1} >>>>>>\".format(monitor,\n",
    "                     arr_scores[best_score_idx]),file=sys.stderr)\n",
    "            if len(arr_scores)-best_score_idx>patience:\n",
    "                print(\"<<<<<< {} without improvement in {} epoch, early stopping >>>>>>\".format(\n",
    "                    monitor,patience),file=sys.stderr)\n",
    "                self.net.load_state_dict(torch.load(ckpt_path))\n",
    "                break \n",
    "            \n",
    "        return pd.DataFrame(self.history)\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def evaluate(self, val_data):\n",
    "        val_data = self.accelerator.prepare(val_data)\n",
    "        val_step_runner = StepRunner(net = self.net,stage=\"val\",\n",
    "                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),\n",
    "                    accelerator = self.accelerator)\n",
    "        val_epoch_runner = EpochRunner(val_step_runner)\n",
    "        val_metrics = val_epoch_runner(val_data)\n",
    "        return val_metrics\n",
    "        \n",
    "       \n",
    "    @torch.no_grad()\n",
    "    def predict(self, dataloader):\n",
    "        dataloader = self.accelerator.prepare(dataloader)\n",
    "        result = torch.cat([self.forward(t[0]) for t in dataloader])\n",
    "        return result.data\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32761275",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras.metrics import AUC\n",
    "\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "306a3dc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfhistory = model.fit(train_data=dl_train,val_data=dl_val,epochs=100, patience=5,\n",
    "                      monitor = \"val_auc\",mode=\"max\",ckpt_path='checkpoint.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9f5a1b8",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wiwm0e3gj20mn0f8goa.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97bac7d",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a577c97",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'svg'\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_metric(dfhistory, metric):\n",
    "    train_metrics = dfhistory[\"train_\"+metric]\n",
    "    val_metrics = dfhistory['val_'+metric]\n",
    "    epochs = range(1, len(train_metrics) + 1)\n",
    "    plt.plot(epochs, train_metrics, 'bo--')\n",
    "    plt.plot(epochs, val_metrics, 'ro-')\n",
    "    plt.title('Training and validation '+ metric)\n",
    "    plt.xlabel(\"Epochs\")\n",
    "    plt.ylabel(metric)\n",
    "    plt.legend([\"train_\"+metric, 'val_'+metric])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1b52108",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_metric(dfhistory,\"loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4edc1d36",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wizlzzsij20fv0a73yt.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acfd031e",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_metric(dfhistory,\"auc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f540023",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h2wiymw5nzj20fj0aaaad.jpg) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a8cb4cd",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f99aaf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "preds = torch.sigmoid(model.predict(dl_val))\n",
    "labels = torch.cat([x[-1] for x in dl_val])\n",
    "\n",
    "val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())\n",
    "print(val_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a79be59",
   "metadata": {},
   "source": [
    "0.7820486224544348\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d139d55",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a3cc987",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.net.state_dict(),\"best_dcn.pt\")\n",
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(\"best_dcn.pt\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cae86f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "net_clone.eval()\n",
    "preds = torch.cat([torch.sigmoid(net_clone(x[0])).data for x in dl_val]) \n",
    "labels = torch.cat([x[-1] for x in dl_val])\n",
    "\n",
    "val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())\n",
    "print(val_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd06a8e",
   "metadata": {},
   "source": [
    "0.7820486196785761\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac90f13",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}