rtf_test_for_zipformer_transducer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPCRupd5RFHchDmVZ3KucSK",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/k2-fsa/colab/blob/master/rtf_test_for_zipformer_transducer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "\n",
        "\n",
        "This colab notebook tests the RTF ([Real-time factor](https://openvoice-tech.net/index.php/Real-time-factor))\n",
        "of the [Zipformer][zipformer] transducer model from [icefall][icefall].\n",
        "\n",
        "For the CPU test, we use [sherpa-onnx][sherpa-onnx]\n",
        "\n",
        "For the GPU test, we infer the RTF from the decoding logs for the librispeech test-clean and test-other dataset.\n",
        "\n",
        "[icefall]: https://github.com/k2-fsa/icefall\n",
        "\n",
        "[sherpa-onnx]: https://github.com/k2-fsa/sherpa-onnx\n",
        "\n",
        "[zipformer]: https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/zipformer\n",
        "\n"
      ],
      "metadata": {
        "id": "SOrRqJX0vXwd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The results are summarized in the following table:\n",
        "\n",
        "\n",
        "## CPU RTF\n",
        "\n",
        "Use CPU in this colab notebook:\n",
        "\n",
        "|model type| weight type|# parameters (M)| RTF|\n",
        "|---|---|---|---|\n",
        "|non-streaming|**float32**| [65.55](https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md#non-streaming-1)|0.064|\n",
        "|non-streaming|**int8**|[65.55](https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md#non-streaming-1)|0.056|\n",
        "|streaming|**float32**|[66.11](https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md#normal-scaled-model-number-of-model-parameters-66110931-ie-6611-m)|0.18|\n",
        "|streaming|**int8**|[66.11](https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md#normal-scaled-model-number-of-model-parameters-66110931-ie-6611-m)|0.13|\n",
        "\n",
        "## GPU RTF\n",
        "\n",
        "Use V100 with 32GB RAM:\n",
        "\n",
        "|model type| dataset | RTF|\n",
        "|---|---|---|\n",
        "|non-streaming| test-clean |0.00298|\n",
        "|non-streaming| test-other |0.00321|\n",
        "|streaming| test-clean |0.00895|\n",
        "|streaming| test-other |0.01111|"
      ],
      "metadata": {
        "id": "8vz2BPAa2fHE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Display CPU information"
      ],
      "metadata": {
        "id": "kLWHEn1SvexF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "\n",
        "lscpu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IESEIvUOvdmH",
        "outputId": "b46016ba-052a-43d1-a5a6-3ea23fea780f"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Architecture:            x86_64\n",
            "  CPU op-mode(s):        32-bit, 64-bit\n",
            "  Address sizes:         46 bits physical, 48 bits virtual\n",
            "  Byte Order:            Little Endian\n",
            "CPU(s):                  2\n",
            "  On-line CPU(s) list:   0,1\n",
            "Vendor ID:               GenuineIntel\n",
            "  Model name:            Intel(R) Xeon(R) CPU @ 2.20GHz\n",
            "    CPU family:          6\n",
            "    Model:               79\n",
            "    Thread(s) per core:  2\n",
            "    Core(s) per socket:  1\n",
            "    Socket(s):           1\n",
            "    Stepping:            0\n",
            "    BogoMIPS:            4399.99\n",
            "    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc\n",
            "                         a cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscal\n",
            "                         l nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopo\n",
            "                         logy nonstop_tsc cpuid tsc_known_freq pni pclmulqdq sss\n",
            "                         e3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes \n",
            "                         xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefe\n",
            "                         tch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_ad\n",
            "                         just bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed ad\n",
            "                         x smap xsaveopt arat md_clear arch_capabilities\n",
            "Virtualization features: \n",
            "  Hypervisor vendor:     KVM\n",
            "  Virtualization type:   full\n",
            "Caches (sum of all):     \n",
            "  L1d:                   32 KiB (1 instance)\n",
            "  L1i:                   32 KiB (1 instance)\n",
            "  L2:                    256 KiB (1 instance)\n",
            "  L3:                    55 MiB (1 instance)\n",
            "NUMA:                    \n",
            "  NUMA node(s):          1\n",
            "  NUMA node0 CPU(s):     0,1\n",
            "Vulnerabilities:         \n",
            "  Itlb multihit:         Not affected\n",
            "  L1tf:                  Mitigation; PTE Inversion\n",
            "  Mds:                   Vulnerable; SMT Host state unknown\n",
            "  Meltdown:              Vulnerable\n",
            "  Mmio stale data:       Vulnerable\n",
            "  Retbleed:              Vulnerable\n",
            "  Spec store bypass:     Vulnerable\n",
            "  Spectre v1:            Vulnerable: __user pointer sanitization and usercopy ba\n",
            "                         rriers only; no swapgs barriers\n",
            "  Spectre v2:            Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBR\n",
            "                         S: Not affected\n",
            "  Srbds:                 Not affected\n",
            "  Tsx async abort:       Vulnerable\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install sherpa-onnx"
      ],
      "metadata": {
        "id": "KGHOoZwEviHf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install sherpa-onnx"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tz4dMpz-vNKt",
        "outputId": "f4a494f3-7bd0-461d-bee2-c6c94aa52e4b"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting sherpa-onnx\n",
            "  Downloading sherpa_onnx-1.7.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m31.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sherpa-onnx) (1.23.5)\n",
            "Collecting sentencepiece==0.1.96 (from sherpa-onnx)\n",
            "  Downloading sentencepiece-0.1.96-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: sentencepiece, sherpa-onnx\n",
            "Successfully installed sentencepiece-0.1.96 sherpa-onnx-1.7.9\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download the non-streaming zipformer pre-trained model"
      ],
      "metadata": {
        "id": "zPVnxMPYvvB-"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8WWx4ZS_vI4C",
        "outputId": "2522253d-8c5f-4524-ebd1-3a826c28af41"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'sherpa-onnx-zipformer-en-2023-06-26'...\n",
            "remote: Enumerating objects: 20, done.\u001b[K\n",
            "remote: Total 20 (delta 0), reused 0 (delta 0), pack-reused 20\u001b[K\n",
            "Unpacking objects: 100% (20/20), 666.78 KiB | 6.67 MiB/s, done.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "%%shell\n",
        "\n",
        "# Please see https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/zipformer-transducer-models.html#csukuangfj-sherpa-onnx-zipformer-en-2023-06-26-english\n",
        "\n",
        "GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/csukuangfj/sherpa-onnx-zipformer-en-2023-06-26\n",
        "cd sherpa-onnx-zipformer-en-2023-06-26\n",
        "git lfs pull --include \"*.onnx\""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## float32 model test"
      ],
      "metadata": {
        "id": "jICBRc-_v4Fa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "\n",
        "sherpa-onnx-offline \\\n",
        "  --tokens=./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt \\\n",
        "  --encoder=./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.onnx \\\n",
        "  --decoder=./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.onnx \\\n",
        "  --joiner=./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.onnx \\\n",
        "  --num-threads=1 \\\n",
        "  --decoding-method=greedy_search \\\n",
        "  ./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k8c2S32Iv15j",
        "outputId": "7ff7e4ab-3b0c-419e-bd47-c029dfbd82d8"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/project/sherpa-onnx/csrc/parse-options.cc:Read:361 sherpa-onnx-offline --tokens=./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt --encoder=./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.onnx --decoder=./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.onnx --joiner=./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.onnx --num-threads=1 --decoding-method=greedy_search ./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav \n",
            "\n",
            "OfflineRecognizerConfig(feat_config=OfflineFeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=OfflineModelConfig(transducer=OfflineTransducerModelConfig(encoder_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.onnx\", decoder_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.onnx\", joiner_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.onnx\"), paraformer=OfflineParaformerModelConfig(model=\"\"), nemo_ctc=OfflineNemoEncDecCtcModelConfig(model=\"\"), whisper=OfflineWhisperModelConfig(encoder=\"\", decoder=\"\", language=\"\", task=\"transcribe\"), tdnn=OfflineTdnnModelConfig(model=\"\"), tokens=\"./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt\", num_threads=1, debug=False, provider=\"cpu\", model_type=\"\"), lm_config=OfflineLMConfig(model=\"\", scale=0.5), decoding_method=\"greedy_search\", max_active_paths=4, context_score=1.5)\n",
            "Creating recognizer ...\n",
            "Started\n",
            "Done!\n",
            "\n",
            "./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav\n",
            "{\"text\":\" GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONORED BOSOM TO CONNECT HER PARENT FOREVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\",\"timestamps\":\"[0.00, 0.24, 0.56, 0.76, 0.92, 1.04, 1.16, 1.20, 1.36, 1.52, 1.64, 1.80, 1.88, 2.00, 2.16, 2.32, 2.40, 2.64, 2.88, 3.12, 3.24, 3.48, 3.56, 3.72, 3.92, 4.12, 4.40, 4.52, 4.72, 4.96, 5.16, 5.36, 5.64, 6.12, 6.28, 6.52, 6.88, 7.12, 7.32, 7.56, 7.92, 8.16, 8.28, 8.40, 8.48, 8.64, 8.76, 8.88, 9.04, 9.28, 9.44, 9.52, 9.60, 9.72, 9.92, 9.96, 10.16, 10.48, 10.72, 10.80, 11.04, 11.20, 11.36, 11.56, 11.76, 12.00, 12.12, 12.28, 12.32, 12.52, 12.72, 12.84, 12.92, 13.04, 13.20, 13.44, 13.68, 13.84, 14.00, 14.16, 14.28, 14.40, 14.56, 14.72, 14.76, 15.00, 15.28, 15.48, 15.68, 15.96, 16.16, 16.20, 16.28]\",\"tokens\":[\" GO\",\"D\",\" AS\",\" A\",\" DI\",\"RE\",\"C\",\"T\",\" CON\",\"SE\",\"QUE\",\"N\",\"CE\",\" OF\",\" THE\",\" S\",\"IN\",\" WHICH\",\" MAN\",\" TH\",\"US\",\" P\",\"UN\",\"ISH\",\"ED\",\" HAD\",\" GIVE\",\"N\",\" HER\",\" A\",\" LOVE\",\"LY\",\" CHILD\",\" WHO\",\"SE\",\" PLACE\",\" WAS\",\" ON\",\" THAT\",\" SAME\",\" DIS\",\"HO\",\"N\",\"OR\",\"ED\",\" BO\",\"S\",\"OM\",\" TO\",\" CON\",\"NE\",\"C\",\"T\",\" HER\",\" P\",\"AR\",\"ENT\",\" FOR\",\"E\",\"VER\",\" WITH\",\" THE\",\" RA\",\"CE\",\" AND\",\" DE\",\"S\",\"C\",\"ENT\",\" OF\",\" MO\",\"R\",\"T\",\"AL\",\"S\",\" AND\",\" TO\",\" BE\",\" FI\",\"N\",\"AL\",\"LY\",\" A\",\" B\",\"LESS\",\"ED\",\" SO\",\"UL\",\" IN\",\" HE\",\"A\",\"VE\",\"N\"]}\n",
            "----\n",
            "num threads: 1\n",
            "decoding method: greedy_search\n",
            "Elapsed seconds: 1.062 s\n",
            "Real time factor (RTF): 1.062 / 16.715 = 0.064\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## int8 model test"
      ],
      "metadata": {
        "id": "VOovcYj0v77A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "sherpa-onnx-offline \\\n",
        "  --tokens=./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt \\\n",
        "  --encoder=./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.int8.onnx \\\n",
        "  --decoder=./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.int8.onnx \\\n",
        "  --joiner=./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.int8.onnx \\\n",
        "  --num-threads=1 \\\n",
        "  --decoding-method=greedy_search \\\n",
        "  ./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uqOwzMqkv6FJ",
        "outputId": "b4637483-8b4b-4eab-cbf4-1ec103a040f2"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/project/sherpa-onnx/csrc/parse-options.cc:Read:361 sherpa-onnx-offline --tokens=./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt --encoder=./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.int8.onnx --decoder=./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.int8.onnx --joiner=./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.int8.onnx --num-threads=1 --decoding-method=greedy_search ./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav \n",
            "\n",
            "OfflineRecognizerConfig(feat_config=OfflineFeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=OfflineModelConfig(transducer=OfflineTransducerModelConfig(encoder_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/encoder-epoch-99-avg-1.int8.onnx\", decoder_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/decoder-epoch-99-avg-1.int8.onnx\", joiner_filename=\"./sherpa-onnx-zipformer-en-2023-06-26/joiner-epoch-99-avg-1.int8.onnx\"), paraformer=OfflineParaformerModelConfig(model=\"\"), nemo_ctc=OfflineNemoEncDecCtcModelConfig(model=\"\"), whisper=OfflineWhisperModelConfig(encoder=\"\", decoder=\"\", language=\"\", task=\"transcribe\"), tdnn=OfflineTdnnModelConfig(model=\"\"), tokens=\"./sherpa-onnx-zipformer-en-2023-06-26/tokens.txt\", num_threads=1, debug=False, provider=\"cpu\", model_type=\"\"), lm_config=OfflineLMConfig(model=\"\", scale=0.5), decoding_method=\"greedy_search\", max_active_paths=4, context_score=1.5)\n",
            "Creating recognizer ...\n",
            "Started\n",
            "Done!\n",
            "\n",
            "./sherpa-onnx-zipformer-en-2023-06-26/test_wavs/1.wav\n",
            "{\"text\":\" GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONORED BOSOM TO CONNECT HER PARENT FOREVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\",\"timestamps\":\"[0.00, 0.24, 0.56, 0.76, 0.92, 1.04, 1.16, 1.20, 1.36, 1.52, 1.64, 1.80, 1.88, 2.00, 2.16, 2.32, 2.40, 2.64, 2.88, 3.12, 3.24, 3.48, 3.52, 3.72, 3.92, 4.12, 4.40, 4.52, 4.72, 4.96, 5.12, 5.40, 5.64, 6.12, 6.28, 6.52, 6.88, 7.12, 7.32, 7.56, 7.92, 8.16, 8.28, 8.40, 8.48, 8.64, 8.76, 8.88, 9.04, 9.28, 9.44, 9.52, 9.60, 9.72, 9.92, 9.96, 10.16, 10.48, 10.72, 10.80, 11.04, 11.20, 11.36, 11.56, 11.76, 12.00, 12.12, 12.28, 12.32, 12.52, 12.72, 12.84, 12.92, 13.04, 13.20, 13.44, 13.68, 13.84, 14.00, 14.16, 14.28, 14.40, 14.56, 14.72, 14.76, 15.00, 15.28, 15.48, 15.68, 15.96, 16.16, 16.20, 16.28]\",\"tokens\":[\" GO\",\"D\",\" AS\",\" A\",\" DI\",\"RE\",\"C\",\"T\",\" CON\",\"SE\",\"QUE\",\"N\",\"CE\",\" OF\",\" THE\",\" S\",\"IN\",\" WHICH\",\" MAN\",\" TH\",\"US\",\" P\",\"UN\",\"ISH\",\"ED\",\" HAD\",\" GIVE\",\"N\",\" HER\",\" A\",\" LOVE\",\"LY\",\" CHILD\",\" WHO\",\"SE\",\" PLACE\",\" WAS\",\" ON\",\" THAT\",\" SAME\",\" DIS\",\"HO\",\"N\",\"OR\",\"ED\",\" BO\",\"S\",\"OM\",\" TO\",\" CON\",\"NE\",\"C\",\"T\",\" HER\",\" P\",\"AR\",\"ENT\",\" FOR\",\"E\",\"VER\",\" WITH\",\" THE\",\" RA\",\"CE\",\" AND\",\" DE\",\"S\",\"C\",\"ENT\",\" OF\",\" MO\",\"R\",\"T\",\"AL\",\"S\",\" AND\",\" TO\",\" BE\",\" FI\",\"N\",\"AL\",\"LY\",\" A\",\" B\",\"LESS\",\"ED\",\" SO\",\"UL\",\" IN\",\" HE\",\"A\",\"VE\",\"N\"]}\n",
            "----\n",
            "num threads: 1\n",
            "decoding method: greedy_search\n",
            "Elapsed seconds: 0.928 s\n",
            "Real time factor (RTF): 0.928 / 16.715 = 0.056\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download the streaming zipformer pre-trained model"
      ],
      "metadata": {
        "id": "sPJzNDVYwGzo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "\n",
        "# Please see\n",
        "# https://k2-fsa.github.io/sherpa/onnx/pretrained_models/online-transducer/zipformer-transducer-models.html#csukuangfj-sherpa-onnx-streaming-zipformer-en-2023-06-26-english\n",
        "\n",
        "GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26\n",
        "cd sherpa-onnx-streaming-zipformer-en-2023-06-26\n",
        "git lfs pull --include \"*.onnx\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w8LuG8Fqv9ya",
        "outputId": "ff78ff46-023d-4a42-d967-fa0b7575980e"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'sherpa-onnx-streaming-zipformer-en-2023-06-26'...\n",
            "remote: Enumerating objects: 27, done.\u001b[K\n",
            "remote: Total 27 (delta 0), reused 0 (delta 0), pack-reused 27\u001b[K\n",
            "Unpacking objects: 100% (27/27), 667.63 KiB | 6.13 MiB/s, done.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## float32 model test"
      ],
      "metadata": {
        "id": "2BY8N4X9wO2Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "sherpa-onnx \\\n",
        "  --tokens=./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt \\\n",
        "  --encoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.onnx \\\n",
        "  --decoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.onnx \\\n",
        "  --joiner=./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.onnx \\\n",
        "  ./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XvZvoYxewNYh",
        "outputId": "bb987442-706e-4446-d384-e4fb53af07b1"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/project/sherpa-onnx/csrc/parse-options.cc:Read:361 sherpa-onnx --tokens=./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt --encoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.onnx --decoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.onnx --joiner=./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.onnx ./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav \n",
            "\n",
            "OnlineRecognizerConfig(feat_config=FeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=OnlineModelConfig(transducer=OnlineTransducerModelConfig(encoder=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.onnx\", decoder=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.onnx\", joiner=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.onnx\"), paraformer=OnlineParaformerModelConfig(encoder=\"\", decoder=\"\"), tokens=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt\", num_threads=1, debug=False, provider=\"cpu\", model_type=\"\"), lm_config=OnlineLMConfig(model=\"\", scale=0.5), endpoint_config=EndpointConfig(rule1=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=2.4, min_utterance_length=0), rule2=EndpointRule(must_contain_nonsilence=True, min_trailing_silence=1.2, min_utterance_length=0), rule3=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=0, min_utterance_length=20)), enable_endpoint=True, max_active_paths=4, context_score=1.5, decoding_method=\"greedy_search\")\n",
            "./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav\n",
            "Elapsed seconds: 3.1, Real time factor (RTF): 0.18\n",
            " GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONOURED BOSOM TO CONNECT HER PARENT FOR EVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\n",
            "{\"is_final\":false,\"segment\":0,\"start_time\":0.0,\"text\":\" GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONOURED BOSOM TO CONNECT HER PARENT FOR EVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\",\"timestamps\":\"[0.64, 0.76, 1.00, 1.28, 1.36, 1.44, 1.56, 1.60, 1.76, 1.96, 2.08, 2.20, 2.32, 2.44, 2.60, 2.72, 2.76, 3.00, 3.28, 3.52, 3.64, 3.88, 3.92, 4.12, 4.36, 4.56, 4.80, 4.92, 5.12, 5.32, 5.52, 5.80, 6.04, 6.56, 6.76, 6.96, 7.28, 7.56, 7.80, 8.04, 8.36, 8.56, 8.68, 8.76, 8.84, 9.00, 9.16, 9.32, 9.60, 9.68, 9.84, 9.92, 10.00, 10.08, 10.32, 10.36, 10.52, 10.92, 11.16, 11.24, 11.52, 11.64, 11.84, 12.00, 12.20, 12.48, 12.56, 12.64, 12.68, 12.92, 13.16, 13.24, 13.32, 13.40, 13.64, 13.88, 14.08, 14.24, 14.44, 14.56, 14.68, 14.80, 15.04, 15.16, 15.20, 15.48, 15.68, 15.88, 16.08, 16.28, 16.44, 16.48, 16.56]\",\"tokens\":[\" GO\",\"D\",\" AS\",\" A\",\" DI\",\"RE\",\"C\",\"T\",\" CON\",\"SE\",\"QUE\",\"N\",\"CE\",\" OF\",\" THE\",\" S\",\"IN\",\" WHICH\",\" MAN\",\" TH\",\"US\",\" P\",\"UN\",\"ISH\",\"ED\",\" HAD\",\" GIVE\",\"N\",\" HER\",\" A\",\" LOVE\",\"LY\",\" CHILD\",\" WHO\",\"SE\",\" PLACE\",\" WAS\",\" ON\",\" THAT\",\" SAME\",\" DIS\",\"HO\",\"N\",\"OUR\",\"ED\",\" BO\",\"S\",\"OM\",\" TO\",\" CON\",\"NE\",\"C\",\"T\",\" HER\",\" P\",\"AR\",\"ENT\",\" FOR\",\" E\",\"VER\",\" WITH\",\" THE\",\" RA\",\"CE\",\" AND\",\" DE\",\"S\",\"C\",\"ENT\",\" OF\",\" MO\",\"R\",\"T\",\"AL\",\"S\",\" AND\",\" TO\",\" BE\",\" FI\",\"N\",\"AL\",\"LY\",\" A\",\" B\",\"LESS\",\"ED\",\" SO\",\"UL\",\" IN\",\" HE\",\"A\",\"VE\",\"N\"]}\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## int8 model test"
      ],
      "metadata": {
        "id": "PA-5KaNcwlQU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%shell\n",
        "sherpa-onnx \\\n",
        "  --tokens=./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt \\\n",
        "  --encoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx \\\n",
        "  --decoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx \\\n",
        "  --joiner=./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx \\\n",
        "  ./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FIS4MqLBwQWF",
        "outputId": "3db669de-9fcb-4cef-9d1a-97cae44ff6c6"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/project/sherpa-onnx/csrc/parse-options.cc:Read:361 sherpa-onnx --tokens=./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt --encoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx --decoder=./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx --joiner=./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx ./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav \n",
            "\n",
            "OnlineRecognizerConfig(feat_config=FeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=OnlineModelConfig(transducer=OnlineTransducerModelConfig(encoder=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx\", decoder=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx\", joiner=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx\"), paraformer=OnlineParaformerModelConfig(encoder=\"\", decoder=\"\"), tokens=\"./sherpa-onnx-streaming-zipformer-en-2023-06-26/tokens.txt\", num_threads=1, debug=False, provider=\"cpu\", model_type=\"\"), lm_config=OnlineLMConfig(model=\"\", scale=0.5), endpoint_config=EndpointConfig(rule1=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=2.4, min_utterance_length=0), rule2=EndpointRule(must_contain_nonsilence=True, min_trailing_silence=1.2, min_utterance_length=0), rule3=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=0, min_utterance_length=20)), enable_endpoint=True, max_active_paths=4, context_score=1.5, decoding_method=\"greedy_search\")\n",
            "./sherpa-onnx-streaming-zipformer-en-2023-06-26/test_wavs/1.wav\n",
            "Elapsed seconds: 2.2, Real time factor (RTF): 0.13\n",
            " GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONOURED BOSOM TO CONNECT HER PARENT FOR EVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\n",
            "{\"is_final\":false,\"segment\":0,\"start_time\":0.0,\"text\":\" GOD AS A DIRECT CONSEQUENCE OF THE SIN WHICH MAN THUS PUNISHED HAD GIVEN HER A LOVELY CHILD WHOSE PLACE WAS ON THAT SAME DISHONOURED BOSOM TO CONNECT HER PARENT FOR EVER WITH THE RACE AND DESCENT OF MORTALS AND TO BE FINALLY A BLESSED SOUL IN HEAVEN\",\"timestamps\":\"[0.64, 0.76, 1.00, 1.28, 1.36, 1.44, 1.56, 1.60, 1.76, 1.96, 2.08, 2.20, 2.32, 2.44, 2.60, 2.72, 2.76, 3.00, 3.28, 3.52, 3.64, 3.88, 3.92, 4.12, 4.36, 4.56, 4.80, 4.92, 5.12, 5.32, 5.52, 5.80, 6.04, 6.56, 6.76, 6.96, 7.28, 7.56, 7.80, 8.04, 8.40, 8.56, 8.68, 8.76, 8.84, 9.00, 9.16, 9.32, 9.60, 9.68, 9.80, 9.92, 10.00, 10.08, 10.32, 10.36, 10.52, 10.92, 11.16, 11.24, 11.52, 11.64, 11.84, 12.00, 12.20, 12.48, 12.56, 12.68, 12.72, 12.92, 13.16, 13.24, 13.32, 13.40, 13.64, 13.88, 14.08, 14.24, 14.44, 14.56, 14.68, 14.80, 15.04, 15.16, 15.20, 15.48, 15.68, 15.88, 16.08, 16.28, 16.44, 16.48, 16.56]\",\"tokens\":[\" GO\",\"D\",\" AS\",\" A\",\" DI\",\"RE\",\"C\",\"T\",\" CON\",\"SE\",\"QUE\",\"N\",\"CE\",\" OF\",\" THE\",\" S\",\"IN\",\" WHICH\",\" MAN\",\" TH\",\"US\",\" P\",\"UN\",\"ISH\",\"ED\",\" HAD\",\" GIVE\",\"N\",\" HER\",\" A\",\" LOVE\",\"LY\",\" CHILD\",\" WHO\",\"SE\",\" PLACE\",\" WAS\",\" ON\",\" THAT\",\" SAME\",\" DIS\",\"HO\",\"N\",\"OUR\",\"ED\",\" BO\",\"S\",\"OM\",\" TO\",\" CON\",\"NE\",\"C\",\"T\",\" HER\",\" P\",\"AR\",\"ENT\",\" FOR\",\" E\",\"VER\",\" WITH\",\" THE\",\" RA\",\"CE\",\" AND\",\" DE\",\"S\",\"C\",\"ENT\",\" OF\",\" MO\",\"R\",\"T\",\"AL\",\"S\",\" AND\",\" TO\",\" BE\",\" FI\",\"N\",\"AL\",\"LY\",\" A\",\" B\",\"LESS\",\"ED\",\" SO\",\"UL\",\" IN\",\" HE\",\"A\",\"VE\",\"N\"]}\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": []
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# RTF on GPU"
      ],
      "metadata": {
        "id": "Pgs3ukzO6_yr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you are curious about the RTF on GPU, we can compute a rough number from the decoding logs.\n",
        "\n",
        "## non-streaming\n",
        "\n",
        "For instance, for the non-streaming model used above, its decoding logs for the librispeech test-clean and test-other can be found at\n",
        "https://huggingface.co/Zengwei/icefall-asr-librispeech-zipformer-2023-05-15/blob/main/decoding_result/greedy_search/log-decode-epoch-30-avg-9-context-2-max-sym-per-frame-1-use-averaged-model-2023-05-15-19-37-19\n",
        "\n",
        "From the logs, we can get the following data:\n",
        "\n",
        "||start | end| duration (s)|\n",
        "|---|---|---|---|\n",
        "|test-clean|19:37:39|19:38:08|29|\n",
        "|test-other|19:38:10|19:38:36|26|\n",
        "\n",
        "The test-clean dataset has 2 hours 42 minutes of data, while the test-other dataset has 2 hours 15 minutes of data.\n",
        "\n",
        "So the RTF for test clean is:\n",
        "```\n",
        "29 / (2 hours 42 minutes) = 29 / 9720 = 0.00298\n",
        "```\n",
        "\n",
        "The RTF for test-other is\n",
        "```\n",
        "26 / (2 hours 15 minutes) = 26 / 8100 = 0.00321\n",
        "```\n",
        "\n",
        "Note that the GPU used during decoding is V100 with 32GB RAM.\n",
        "\n",
        "## streaming\n",
        "\n",
        "For the streaming model, the decoding logs can be found at\n",
        "https://huggingface.co/Zengwei/icefall-asr-librispeech-streaming-zipformer-2023-05-17/blob/main/decoding_results/streaming/greedy_seearch/log-decode-epoch-30-avg-8-chunk-16-left-context-128-use-averaged-model-2023-05-12-20-46-34\n",
        "\n",
        "From the logs, we can get the following data:\n",
        "\n",
        "||start | end| duration (s)|\n",
        "|---|---|---|---|\n",
        "|test-clean|20:46:41|20:48:08|87|\n",
        "|test-other|20:48:08|20:49:38|90|\n",
        "\n",
        "So the RTF for test clean is:\n",
        "```\n",
        "87 / (2 hours 42 minutes) = 87 / 9720 = 0.00895\n",
        "```\n",
        "\n",
        "The RTF for test-other is\n",
        "```\n",
        "90 / (2 hours 15 minutes) = 90 / 8100 = 0.01111\n",
        "```\n",
        "\n",
        "Note that the GPU used during decoding is V100 with 32GB RAM.\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "The RTFs are summarized in the following table:\n",
        "\n",
        "|model type| dataset | RTF|\n",
        "|---|---|---|\n",
        "|non-streaming| test-clean |0.00298|\n",
        "|non-streaming| test-other |0.00321|\n",
        "|streaming| test-clean |0.00895|\n",
        "|streaming| test-other |0.01111|\n",
        "\n",
        "**Note**: We use PyTorch with GPU during decoding the librispeech test dataset.\n",
        "\n"
      ],
      "metadata": {
        "id": "b0bt4NrL7CDg"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "uAok2Pdywr1O"
      },
      "execution_count": 8,
      "outputs": []
    }
  ]
}