encoder.json does not exist in chunks directory.

[henryhu712]

There is an error during building in Next.js applications:

info  - Collecting page data ...Error: ENOENT: no such file or directory, open '/home/hh/backre-com/.next/server/chunks/encoder.json'

I tried Node.js 16 and 18, result in the same error.

[IanzolaLegend]

I have the same issue, i'm using node 16

[nbeny]

same in lambda function

[Rintaro-Shimada]

Same problem.

I am using Electron from Typescript and Webpack.
when build process. Mac OS can build, Windows can't

Version
node: v18.16.0

[sumith-lawpath]

Same problem.
I'm using esbuild and node v18.15.0

[SealinGp]

node: v16.16.0
environment: macos
dev: vscode plugin dev
same issue:

Activating extension 'undefined_publisher.x' failed: ENOENT: no such file or directory, open '/Users/Public/dev/z-copilot/dist/encoder.json'.

[pfeff]

Running into this problem also in an Electron App (Obsidian Plugin). It looks like the problem might be coming from around line 5 where its trying to load encoder.json using __dirname. This is resolving to /Applications/Obsidian.app/Contents/Resources/electron.asar/renderer/encoder.json instead of something relative to node_modules.

[granmoe]

It seems that this file is somehow not getting bundled. I worked around this by manually copying it into the build like this:

(next.config.js)

const CopyPlugin = require('copy-webpack-plugin');

/** @type {import('next').NextConfig} */
module.exports = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      config.plugins.push(
        new CopyPlugin({
          patterns: [
            {
              from: 'node_modules/gpt-3-encoder/encoder.json',
              to: 'encoder.json',
            },
            {
              from: 'node_modules/gpt-3-encoder/vocab.bpe',
              to: 'vocab.bpe',
            },
          ],
        })
      );
    }
    return config;
  },
};

[granmoe]

I ended up switching my library, GPT Toolkit, to gpt-tokenizer which does not have this issue.