Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(core): Move queue recovery to scaling service (no-changelog) #10368

Merged
merged 7 commits into from
Aug 13, 2024

Conversation

ivov
Copy link
Contributor

@ivov ivov commented Aug 13, 2024

Queue recovery belongs in the scaling service, which did not exist when queue recovery was created.

@n8n-assistant n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels Aug 13, 2024
tomi
tomi previously approved these changes Aug 13, 2024
Copy link
Contributor

@tomi tomi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀 Couple minor comments/questions

packages/cli/src/scaling/scaling.service.ts Outdated Show resolved Hide resolved
Comment on lines +301 to +304
if (storedIds.length === 0) {
this.logger.debug('[ScalingService] Completed queue recovery check, no dangling executions');
return waitMs;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getInProgressExecutionIds seems to get executions with status new and running, but not waiting. If there are only waiting executions in DB, they are not checked as we short-circuit here. Is that intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Queue recovery reconciles Redis state with DB state, so if Redis knows an execution not to be in any state in the Bull queue anymore, then the DB status is stale and needs to be overridden to crashed.

In your example, if a DB has zero new or running executions, e.g. all waiting, then those waiting executions have not made it to the Bull queue yet, so there is nothing to reconcile. The requirements in the original story mentioned only new and running.

Please let me know if this is what you meant.

packages/cli/src/scaling/types.ts Show resolved Hide resolved
Copy link
Contributor

⚠️ Some Cypress E2E specs are failing, please fix them before merging

Copy link

cypress bot commented Aug 13, 2024



Test summary

397 0 0 0Flakiness 1


Run details

Project n8n
Status Passed
Commit 7b47c75
Started Aug 13, 2024 12:18 PM
Ended Aug 13, 2024 12:23 PM
Duration 04:52 💡
OS Linux Debian -
Browser Electron 118

View run in Cypress Cloud ➡️


Flakiness

e2e/14-mapping.cy.ts Flakiness
1 Data mapping > maps expressions from table header

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Cloud

Copy link
Contributor

@tomi tomi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

Copy link
Contributor

✅ All Cypress E2E specs passed

@ivov ivov merged commit 56c4692 into master Aug 13, 2024
37 checks passed
@ivov ivov deleted the move-queue-recovery-to-scaling-service branch August 13, 2024 13:06
MiloradFilipovic added a commit that referenced this pull request Aug 14, 2024
* master: (98 commits)
  feat(core): Allow overriding npm registry for community packages (#10325)
  feat(core): Upgrade DB drivers (no-changelog) (#10370)
  fix(editor): Fix bug causing workflow debugging to not work in new canvas (no-changelog) (#10384)
  fix: Fix issue with some errors not being handled correctly (no-changelog) (#10371)
  fix(core): Filter out prototype and constructor lookups in expressions (#10382)
  fix(editor): Connect up new project viewer role to the FE (#9913)
  refactor(core): Move queue recovery to scaling service (no-changelog) (#10368)
  fix(core): Account for owner when filtering by project ID in `GET /workflows` in Public API (#10379)
  fix(editor): Fix rendering of SVG icons in public chat on iOS (#10381)
  fix: Require mfa code to disable mfa (#10345)
  ci: Disable turbo cache when running tests for coverage collection (no-changelog) (#10380)
  refactor(editor): Add typed event bus (no-changelog) (#10367)
  refactor(core): Remove unused constants in Redis channels (no-changelog) (#10369)
  fix(editor): Revert change that hid swagger docs in the ui (#10350)
  fix(Okta Node): Add missing codex file (no-changelog) (#10372)
  fix(core): Fix worker shutdown errors when active executions (#10353)
  refactor(core): Rename ActiveWebhooks to LiveWebhooks (no-changelog) (#10355)
  fix(n8n Form Trigger Node): Fix issue preventing v1 node from working (#10364)
  feat(editor): Upgrade markdown-it to address AIKIDO-2024-10034 (no-changelog) (#10358)
  ci: Upgrade axios to address CVE-2024-39338 (no-changelog) (#10365)
  ...

# Conflicts:
#	packages/design-system/package.json
@janober
Copy link
Member

janober commented Aug 15, 2024

Got released with n8n@1.55.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team Released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants