Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prod release #797

Merged
merged 2 commits into from
Nov 21, 2023
Merged

Prod release #797

merged 2 commits into from
Nov 21, 2023

Conversation

snim2
Copy link
Collaborator

@snim2 snim2 commented Nov 20, 2023

  • Set CACHEOPS_DEGRADE_ON_FAILURE to True

Testing

  • List env vars for staging dalmatian service list-environment-variables -i nhsx-website -s web -e staging
  • Set REDIS_HOST to something incorrect
  • Restart the containers dalmatian service restart-containers -s web -e staging -i nhsx-website
  • Keep Papertrail open and watch for errors like RuntimeWarning: The cacheops cache is unreachable!
  • Check that the site is still running
  • Set the REDIS_HOST back to it's previous value
  • Restart the containers again

snim2 and others added 2 commits November 20, 2023 09:21
This commit is a follow-up to Incident 152 (and 95).

Currently, the app fails with HTTP 5xx errors if the associated
Redis cache is unreachable or cannot be used.

This commit sets the `CACHEOPS_DEGRADE_ON_FAILURE` environment
variable to be `True`. This env var instructs django-cacheops
to wrap Redis calls in a try/except block and to handle
connection errors and timeout errors gracefully, by writing
a warning to the logs, rather than letting the relevant
exception propagate up the stack.

See:
    https://github.com/Suor/django-cacheops#setup
    https://github.com/Suor/django-cacheops/blob/master/cacheops/redis.py#L12-L22
…redis-failure

Set CACHEOPS_DEGRADE_ON_FAILURE to True
Copy link
Collaborator

@jkeasley jkeasley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@snim2 snim2 merged commit ac2e8a5 into master Nov 21, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants