ssr-vue client navigation test is flaky

[GrygrFlzr]

Describe the bug

The client navigation test in packages/playground/ssr-vue/__tests__/ssr-vue.spec.ts is flaky and can erroneously fail a test run.

Reproduction

I am always able to replicate this on my system, but I suspect not every system has this issue.
On a fresh clone of the Vite monorepo:

yarn --frozen-lockfile
cd packages/vite && yarn build && cd ../..
cd packages/plugin-vue && yarn build && cd ../..
yarn test

Here's a CI run of it failing despite no changes to Vite: https://app.circleci.com/pipelines/github/vitejs/vite/3295/workflows/5c10a93d-39c3-40cd-becc-56a1c2601d7f/jobs/3190

System Info

• vite version: commit 23f57ee

Local System

• Operating System: Windows 10 10.0.19042
• Node version: 14.16.0
• Package manager: Yarn: 1.22.10

CI System

• Operating System: Ubuntu 18.04.5 LTS
• Node version: 12?
• Package manager: Yarn: 1.22.4

Logs

Summary of all failing tests
 FAIL  packages/playground/ssr-vue/__tests__/ssr-vue.spec.ts
  ● client navigation

    expect(received).toMatch(expected)

    Expected substring: "About"
    Received string:    "changed"

      112 |   await page.click('a[href="/about"]')
      113 |   await page.waitForTimeout(10)
    > 114 |   expect(await page.textContent('h1')).toMatch('About')
          |                                        ^
      115 |   editFile('src/pages/About.vue', (code) => code.replace('About', 'changed'))
      116 |   await untilUpdated(() => page.textContent('h1'), 'changed')
      117 | })

      at Object.<anonymous> (packages/playground/ssr-vue/__tests__/ssr-vue.spec.ts:114:40)

[yyx990803]

Added a "test" label since this isn't a user-affecting bug.

[Shinigami92]

I'm able to reproduce that on my machine!

System Info

Output of npx envinfo --system --npmPackages vite,@vitejs/plugin-vue --binaries --browsers:

  System:
    OS: Linux 5.10 Arch Linux
    CPU: (28) x64 Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz
    Memory: 51.93 GB / 62.49 GB
    Container: Yes
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 15.6.0 - ~/.nvm/versions/node/v15.6.0/bin/node
    Yarn: 1.22.10 - /usr/bin/yarn
    npm: 7.4.0 - ~/.nvm/versions/node/v15.6.0/bin/npm

Used package manager: yarn

Also here and there our CI now can reproduce this (not only the new GHA runners, but also CircleCI and AppVeyor before sometimes failed with that randomly)

Example:

• https://github.com/vitejs/vite/pull/2935/checks?check_run_id=2312688852#step:8:16
  

[brillout]

I had the same problem with vite-plugin-ssr's test suite. I solved it by doing the following.

    // Client-side routing
    await page.click('a[href="/star-wars"]')
    await autoRetry(async () => {
      expect(await page.textContent('h1')).toBe('Star Wars Movies')
    })

Source: https://github.com/brillout/vite-plugin-ssr/blob/ae807f634c2139369b4c804bab22303cb2405c55/examples/.testPages.ts#L36-L40

async function autoRetry(test: () => void | Promise<void>): Promise<void> {
  const timeout = 60 * 1000
  const period = 100
  const numberOfTries = timeout / period
  let i = 0
  while (true) {
    try {
      await test()
      return
    } catch (err) {
      i = i + 1
      if (i > numberOfTries) {
        throw err
      }
    }
    await sleep(period)
  }
}

Source: https://github.com/brillout/vite-plugin-ssr/blob/ae807f634c2139369b4c804bab22303cb2405c55/tests/setup.ts#L150-L167

[brillout]

(It makes sense when you think about it: Jest cannot know when a client-side routing is finished. It's inherently a race condition from Jest's perspective.)