before:spec broken?

[ciprianene]

Current behavior

When trying to run a "before:spec" function, nothing happens.

Desired behavior

No response

Test code to reproduce

setupNodeEvents(on, config) {
on("before:spec", (spec) => {
console.log("Running before spec: ", { spec });
});
}

(//experimentalInteractiveRunEvents is set to true.)

the log will return empty values for the spec object.

Also logged what's returned by before:run and the spec shows as an empty array.

Cypress Version

10.1.0

Other

No response

[rachelruderman]

Hi @ciprianene , thank you for reporting! I put together this repo to reproduce your issue. It looks like you found a buggo! 🐛

Observed behavior:

1. npx cypress open
2. Click "E2E" -> "Start E2E testing in Chrome"
3. Log appears in terminal with empty string for name, relative, absolute properties. User sees spec list in GUI
4. Run a spec. Log does not appear again despite starting, stopping, or switching specs.

Expected behavior:
Log should appear when a spec file is selected, and the values of spec should all be populated

Documentation
https://docs.cypress.io/api/plugins/before-spec-api#Syntax

[ShadowLNC]

Just to add my team's use case here - we are trying to load in some data during test loading (we wrapped Mocha's it call with additional data of files to be loaded - cy.* is not available until inside the it block) and we're passing it to the test via the Cypress config, similar to #22614.

Right now we're loading in all possible required files since we can't get the specific spec during interactive mode, which we anticipate will become a performance bottleneck as our test suite grows (approx 10-15sec per 1000 data files according to my testing).

[BlueWinds]

Interesting that this appears to be in open mode only. Investigating now.

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  // setupNodeEvents can be defined in either
  // the e2e or component configuration
  experimentalInteractiveRunEvents: true,
  e2e: {
    setupNodeEvents(on, config) {
      on('before:spec', (spec) => {
        console.log(`This is the before:spec event`)
        console.log(spec)
      })

      on('before:run', (info) => {
        console.log(`This is the before:run event`)
        console.log('experimentalInteractiveRunEvents', info.config.experimentalInteractiveRunEvents)
        console.log(info.specs)
      })
    }
  }
})

This is quite different from run mode, where all information is properly available:

[BlueWinds]

Ok, so looking into it, this is the intended behavior. From our docs:

The before:spec event fires before a spec file is run. When running cypress via cypress open, the event will fire when the browser launches.

I can see a very good argument that this is dumb behavior - before:spec should fire before we run a spec with details about that spec regardless of mode - but it does seem to be intended, at least to the point where we documented it.

I'm going to move this over to a feature request rather than a bug. That doesn't mean I'm not still working on it though! :)

Correction: It looks to be a regression, since it did work properly in pre-10.x versions of Cypress. Looks like it got overlooked during the upgrade. The difference is that opening the browser used to be synonymous with running a particular spec.

[BlueWinds]

So I'm at a bit of a standstill here. Our docs say that the plugin's before:spec can return a promise which will be awaited. However, my current approach was to tap into mocha events - runner:start and runner:end - and mocha events are synchronous. You can't delay execution based on them.

So my options are...

1. Start over from scratch finding a different approach, taking another 2-3 days on this issue.
2. Make run and open mode behave differently and use two different code paths - in run mode Cypress will wait on your promises, in open mode you cannot. This is literally the worst, having multiple code paths for the same functionality.
3. Breaking change, Make the cypress events match the mocha events (you cannot return promises, they are informational not hooks to adjust Cypress' flow)
4. Make a bigger comment about what I've learned here, unassign myself, go back to working on detached DOM.

[BlueWinds]

So I've stopped working on this for now; Anyone else who wants to take a look is welcome. Also copying a bit of conversation I had with @emilyrohrbough here for future reference - she has good ideas on what events we should hook into, rather than my attempt to listen to mocha run:start / run:stop events.

Emily: I have a couple of ideas, but to double check the facts
only seen in open mode and/or in run mode as well?

Blue: Only seen in open mode. In run mode it works, because the events are currently emitted in modes/run.js - but there's no good equivalent for that in open mode, since runs are started / stopped by UI interactions.
I'd like to replace the run-mode solution with something that works for both though.
Rather than having separate code-paths for each mode.

Emily: Right--- so the mocha events would be too late
Because the spec would have loaded already

Blue: Yeah. That's what I meant when I said "back to the drawing board" - would have to find a different way to hook in.

Emily: but you could hook into the reporter:restart:* events for restarting a spec and could likely add an event that emits when the user selects a spec from the spec list
we hook into `reporter:restart:* to reset runner state and reload the page

Blue Frauenglass: Ah, yeah, that sounds like a good place then.
I don't know literally any of the events here, which was what made this take so long. I was going in blind.

Emily: might at least be worth looking at!
I deep dove into these evens earlier this week / been looking at them for quite some time as I made reporter changes

[cypress-bot]

The code for this is done in cypress-io/cypress#23634, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 10.8.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v10.8.0, please open a new issue.